Server system for processing graphic output and responsively blocking select input commands

ABSTRACT

A video server is configured to provide streaming video to players of computer games over a computing network. The video server can provided video of different games to different players simultaneously. This is accomplished by rendering several video streams in parallel using a single GPU (Graphics Processing Unit). The output of the GPU is provided to graphics processing pipelines that are each associated with a specific client/player and are dynamically allocated as needed. A client qualifier may be used to assure that only clients capable of presenting the streaming video to a player at a minimum level of quality receive the video stream. Video frames provided by the video server optionally include overlays added to the output of the GPU. These overlays can include voice data received from another game player. These overlays may be used to prevent presentation of non-allowed input controls.

CLAIM OF PRIORITY

This application is a continuation of and claims benefit of andpriority, under 35 U.S.C. § 120, to co-pending U.S. patent applicationSer. No. 13/350,216, filed on Jan. 13, 2012, and entitled, “CommandSentinel”. The U.S. patent application Ser. No. 13/350,216 is acontinuation-in-part of U.S. patent application Ser. No. 12/826,489filed Jun. 29, 2010 and entitled “Voice Overlay” (since issued as U.S.Pat. No. 8,888,592 on Nov. 18, 2014), which in turn claims benefit ofand priority, under 35 U.S.C. § 119(e), to U.S. Provisional PatentApplication Ser. No. 61/354,699 filed Jun. 14, 2010 and entitled “VideoGame Overlay”. The U.S. patent application Ser. No. 13/350,216 is acontinuation-in-part of U.S. patent application Ser. No. 12/791,819filed Jun. 1, 2010 and entitled “Qualified Video Delivery,” which inturn claims priority and benefit, under 35 U.S.C. § 119(e), to thefollowing U.S. provisional patent applications:

61/183,035 filed Jun. 1, 2009 and entitled “Game Server Architecture,”

61/183,037 filed Jun. 1, 2009 and entitled “Bufferless H.264 Variant,”

61/183,088 filed Jun. 2, 2009 and entitled “1/0 Level Virtualization,”

61/183,546 filed Jun. 2, 2009 and entitled “Self-Spawning GameEnvironments,”

61/323,354 filed Apr. 12, 2010 and entitled “Artificial Frames,” and

61/345,534 filed May 17, 2010 and entitled “Dynamic Game ServerIncluding Qualifier”.

The U.S. patent application Ser. No. 13/350,216 is acontinuation-in-part of U.S. patent application Ser. No. 11/499,524filed Aug. 3, 2006 entitled “Online Game Customer Lead Generation,” andalso claims priority and benefit, under 35 U.S.C. § 119(e), to thefollowing U.S. provisional patent applications:

61/488,297 filed May 20, 2011 and entitled “Web based Game Controller,”

61/558,376 filed Nov. 10, 2011 and entitled “Command Sentinel,” and

61/558,859 filed Nov. 11, 2011 and entitled “Real-time Watermarking.”

The disclosures of all of the aforementioned applications areincorporated herein by reference in their entirety for all purposes.

RELATED APPLICATIONS

This application is related to U.S. patent applications:

Ser. No. 12/790,948 filed May 31, 2010 and entitled “Bufferless H.264Variant,” and

Ser. No. 12/790,955 filed May 31, 2010 and entitled “Game ExecutionEnvironments.”

The disclosures of all of the aforementioned applications areincorporated herein by reference in their entirety for all purposes.

FIELD OF THE INVENTION

The invention is in the field of audio/video processing and morespecifically in the field of audio/video overlays.

BACKGROUND

There are several models for the execution and rendering of video games.In the simplest model a game is executed and rendered on a computingdevice local to a player of the game. In another model a game state ismaintained at a central server and communicated over a computing networkto a client where rendering of video takes place. This model is commonlyused by multiplayer games in which the game state is dependent on inputsreceived over the computing network from multiple players of the game.In a third model a game state is maintained at a central server thatalso renders video for delivery to clients over a computing network as avideo stream.

SUMMARY

The invention includes a video server system configured to render andprovide game video to one or more clients over a computing network, suchas the internet. This server is optionally configured to serve multiplevideo games to multiple clients. The rendered video is based on thestate of one or more games that can be maintained at the video gameserver system and/or received from a separate game server.

In some of the embodiments, a single graphics processing unit (GPU) isused to render multiple video streams representative of different videogames, and or different viewpoints within a game. For example a singleGPU may be used to render video streams for two different game playersin parallel. Each of these video streams is based on different(independent) viewpoints of the game players and is typically providedto different clients used by each of the game players.

In some embodiments, a shared memory is used by a 3D video driver tostore output of the GPU. This shared memory is accessed by one or moredynamic encoding pipeline. Transfer of video data to and/or from thisshared memory is controlled by virtual 3D video drivers. This controlallows delivery of the output of a shared GPU to a plurality of sharedencoding pipelines in a coordinated manner.

Some embodiments include one or more encoding pipelines configured to bedynamically configured to produce various types of video streams. Thesepipelines are provisioned to meet specific needs of a client. They areoptionally configured to provide more than one type of encoding and tochange the encoding applied to a video stream automatically in responseto a change in the needs of the client.

Various embodiments of the invention include a video server systemcomprising a client qualifier configured to remotely determine if aclient is qualified to receive a video stream configured for display onthe client, the qualification being based on abilities of the client toreceive and display the video stream at least a minimum level ofquality; a video source configured to provide the video stream, and toprovide the video stream only if the client is qualified to receive thevideo stream as determined by the client qualifier; an I/O deviceconfigured for communicating with the client via a communicationnetwork; a storage configured to store video data for use by the videosource or the client qualifier; and an electronic processor configuredto execute software components of the video source or the clientqualifier.

Various embodiments of the invention include a video server systemcomprising an application engine configured to execute a first computerapplication; a graphics processing unit that is configured to generate agraphic output in response to graphics data received from the firstcomputer application and a point of view; command processing logic thatis configured to block communication of an input command to the firstcomputer application by examining input commands received by one or moreinput/output devices before forwarding the input commands to the firstcomputer application, so as to affect the graphic output; an encodingpipeline that is configured to generate an encoded video stream usingthe graphic output, wherein the encoded video stream does not includethe input command that is blocked by the command processing logic; anoutput logic that is configured to communicate the encoded video streamto a remote client via a communication network; and a microprocessorthat is configured to execute the command processing logic at the videoserver system.

Various embodiments of the invention include a game serving systemcomprising a first game application stored on a computer readable mediumand configured to execute a first video game responsive to game commandsreceived from a first client; a second game application stored on acomputer readable medium and configured to execute a second gameinstance responsive to game commands received from a second client;logic configured to execute the first game instance and the second gameinstance in parallel; a first virtual 3D video driver configured toreceive a first game environment from the first game application; asecond virtual 3D video driver configured to received second gameenvironment from the second game application; a 3D video driverconfigured to receive an output of the first and second virtual 3D videodrivers; a graphics processing unit configured to generate a first videostream by rendering the first game environment, and to generate a secondvideo stream by rendering the second game environment, in parallel; afirst encoding pipeline configured to encode the first video stream in afirst codec; a second encoding pipeline configured to encode the secondvideo stream in a second codec; and an output configured to provide thefirst video stream to the first client and the second video stream tothe second client.

Various embodiments of the invention include a game serving systemcomprising a first game application stored on a computer readable mediumand configured to execute a first game instance responsive to gamecommands received from a first client; a second game application storedon a computer readable medium and configured to execute a second gameinstance responsive to game commands received from a second client;logic configured to execute the first game instance and the second gameinstance in parallel; a 3D video driver configured to received differentgame environments from the first game application and from the secondgame application; a graphics processing unit configured to generate afirst video frame by rendering the game environment from the first gameapplication, and to generate a second video frame by rendering the gameenvironment from the second game application; a first encoding pipelineconfigured to encode the first video frame in a first video format; asecond encoding pipeline configured to encode the second video frame ina second video format; a memory shared by at least the 3D video driverand at least the first encoding pipeline; logic configured to transferthe first video frame from the 3D video driver to the shared memory insync with the production of the second video frame; and an outputconfigured to provide the first video stream to the first client and thesecond video stream to the second client.

Various embodiments of the invention include a game serving systemcomprising a first game application stored on a computer readable mediumand configured to execute a first game instance responsive to gamecommands received from a first client; a second game application storedon a computer readable medium and configured to execute a second gameinstance responsive to game commands received from a second client;logic configured to execute the first game instance and the second gameinstance in parallel; logic configured to provision a first encoderpipeline and a second encoder pipeline, the first encoder pipelineincluding a dynamically allocated encoder selected based on needs of thefirst client, the second encoder pipeline including a dynamicallyallocated encoder selected based on needs of the second client; one ormore processor configured to execute the logic configured to spawn afirst encoder pipeline and a second encoder pipeline, the first gameapplication, the second game application; and an output configured toprovide a first video stream generated by the first encoder pipeline tothe first client, and a second video stream generated by the secondencoder pipeline to the second client.

Various embodiments of the invention include a game delivery systemcomprising a plurality of video server systems each configured toprovide streaming video to a plurality of clients over a communicationnetwork, the video server systems being geographically distributed,contents of the streaming video being dependent on game commandsreceived from one or more of the clients and game logic included withineach of the video server system; and a client qualifier configured todetermine if a first of the plurality of clients is qualified to receivethe streaming game video and to determine which of the video serversystem is best qualified to provide the streaming game video to thefirst of the plurality of clients. Optionally, each of the video serversystems is configured to provision a dynamically encoded encodingpipeline responsive to needs of the first of the plurality of clients,and optionally the streaming video is configured to be displayed withina browser. In some embodiments, the determination of which of the videoserver systems is best qualified is dependent on characteristics of acommunication channel between the determined video server system and thefirst of the plurality of clients.

Various embodiments of the invention include a method comprisingreceiving a request from a client to display an object, the object beingconfigured to allow a user of the client to play a video game; testingthe client to determine if the client is qualified to receive astreaming video of the game; qualifying the client based on the testing;selecting a video server system from among a plurality of geographicallydistributed video server systems to provide streaming video to theclient, contents of the streaming video being dependent on game commandsreceived from the client and game logic of the video game included theselected video server system, the selection being based on which of theplurality of video server systems is best qualified to provide thestreaming video to the client. Optionally, the plurality ofgeographically distributed video systems includes video server systemsdisposed in different cities. In some embodiments the method furthercomprises adding a game player to the game and repeating the selectingthe video server system, the selection being further based on which ofthe plurality of video server systems is best qualified to providestreaming video to a client of the added game player, and optionallychanging which of the video server systems provides streaming video tothe client based on the repeated selection.

Various embodiments of the invention include a method of generating gamevideo, the method comprising generating a game environment based on gamerules and game commands received from one or more players; rendering thegame environment to a video frame or audio data; providing the videoframe or audio data to an encoding pipeline; adding an overlay to thevideo frame or audio data; encoding the overlay and the video frame oraudio data; and delivering the encoded overlay and video frame or audiodata to a client of the one or more players via the internet.

Various embodiments of the invention include a game delivery systemcomprising first game logic configured to provide audio data based onevents within a video game; a first encoding pipeline configured toplace an audio overlay on the audio data and to encode the audio overlayand the audio data into an audio packet according to an audio codec; anoutput configured to provide the audio packet to a first client over acommunication network; and an electronic processor configured to executethe first game logic or the encoding pipeline.

Various embodiments of the invention include a game delivery systemcomprising game logic configured to provide a game environment based ongame rules and game commands received from one or more game players; agraphics processing unit configured to render the game environment tovideo data; an encoding pipeline configured to place an overlay on thevideo data and to encode the overlay and the video data as a videoframe; and an output configured to provide the video frame to a clientover a communication network.

Various embodiments of the invention include a method of generating gamevideo, the method comprising generating a game environment based on gamerules and game commands received from one or more players; rendering thegame environment to a video frame or audio data; providing the videoframe or audio data to an encoding pipeline; adding an overlay to thevideo frame or audio data; encoding the overlay and the video frame oraudio data; and delivering the encoded overlay, and video frame or audiodata to a client of the one or more players via the internet.

Various embodiments of the invention include a system comprising firstgame logic configured to provide first audio data based on events withina video game; an audio input configured to receive second audio datafrom a first client; a first encoding pipeline configured to place thesecond audio data as an audio overlay on the first audio data, and toencode the first audio data and the second audio data into a first audiopacket; an output configured to provide the first audio packet to asecond client over a communication network; and an electronic processorconfigured to execute the first game logic or the encoding pipeline.

Various embodiments of the invention include a method of generating gameaudio, the method comprising: generating game audio based on eventswithin a game; receiving audio data from a first player of the game overa computing network; overlaying the game audio and the audio data;encoding the game audio and the audio data to generate an audio stream;and delivering the audio stream including both the game audio and theaudio data to a second player of the game over the computing network.

Various embodiments of the invention include a video server systemcomprising an application engine configured to execute a first computerapplication, a graphics processing unit configured to generate a graphicoutput in response to graphics data received from the first computerapplication, an encoding pipeline configured to generate an encodedvideo stream using the graphic output, sentinel logic configured tomonitor the graphic output and determine that a blocked input control isincluded within the graphic output, the blocked input control beingconfigured to send a blocked command to the first computer application,overlay logic configured to place an overlay on the graphic output, theoverlay configured to change the appearance of the blocked inputcontrol, an output configured to communicate the video stream to aremote client via a communication network, and a processor configured toexecute the sentinel logic, the overlay logic or the command processinglogic.

Various embodiments of the invention include a method of generating gamevideo, the method comprising generating a game environment based on gamerules of a video game and game commands received from one or moreplayers, rendering the game environment to a video frame, providing thevideo frame to an encoding pipeline, monitoring the game environment orthe video frame for the presence of a blocked input control, encodingthe video frame to generate part of a video stream, adding an overlay tothe video frame or the video stream, the overlay being configured tomodify the blocked input control, and delivering the encoded videostream, including the overlay, to a client of the one or more playersvia the internet.

Various embodiments of the invention include a computer readable mediumhaving stored thereon computing instructions configured to monitor agraphics output of a graphics processing unit, computing instructionsconfigured to detect a blocked input control within the graphics output,and computing instructions configured to cause an overlay to be placedon the blocked input control so as to change an appearance of theblocked input control.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a game system, according to variousembodiments of the invention.

FIG. 2 is a block diagram of a video source configured to serve multiplevideo games, according to various embodiments of the invention.

FIG. 3 illustrates methods of receiving a qualified request for a videostream from a client, according to various embodiments of the invention.

FIG. 4 illustrates methods of allocating resources within a videosource, according to various embodiments of the invention.

FIG. 5 illustrates methods of serving multiple video games, according tovarious embodiments of the invention.

FIG. 6 illustrates a game system including more than one video serversystem, according to various embodiments of the invention.

FIG. 7 illustrates methods of providing a video stream to a qualifiedclient, according to various embodiments of the invention.

FIG. 8 illustrates overlays on game video, according to variousembodiments of the invention.

FIGS. 9A, 9B and 9C illustrate game control mappings, according tovarious embodiments of the invention.

FIG. 10 illustrates part of an input map configured for mapping of userinput controls to game commands.

FIG. 11 illustrates methods of using an overlay, according to variousembodiments of the invention.

FIG. 12 illustrates parts of a video server system including sentinellogic, according to various embodiments of the invention.

FIG. 13 illustrates methods of generating game video, according tovarious embodiments of the invention.

FIG. 14 illustrates a method of identifying blocked input controls,according to various embodiments of the invention.

DETAILED DESCRIPTION

FIG.1 is a block diagram of a Game System 100, according to variousembodiments of the invention. Game System 100 is configured to provide avideo stream to one or more Clients 110 via a Network 115. Game System100 typically includes a Video Server System 120 and an optional GameServer 125. Video Server System 120 is configured to provide the videostream to the one or more Clients 110 with a minimal quality of service.For example, Video Server System 120 may receive a game command thatchanges the state of or a point of view within a video game, and provideClients 110 with an updated video stream reflecting this change in statewith minimal lag time. The Video Server System 120 may be configured toprovide the video stream in a wide variety of alternative video formats.Further, the video stream may include video frames configured forpresentation to a user at a wide variety of frame rates. Typical framerates are 30 frames per second, 60 frames per second, and 120 frames persecond. Although higher or lower frame rates are included in alternativeembodiments of the invention.

Clients 110, referred to herein individually as 110A, 110B, etc., mayinclude terminals, personal computers, game consoles, tablet computers,telephones, set top boxes, kiosks, wireless devices, digital pads,stand-alone devices, handheld game playing devices, and/or the like.Typically, Clients 110 are configured to receive encoded video streams,decode the video streams, and present the resulting video to a user,e.g., a player of a game. The processes of receiving encoded videostreams and/or decoding the video streams typically includes storingindividual video frames in a receive buffer of the client. The videostreams may be presented to the user on a display integral to Client 110or on a separate device such as a monitor or television. Clients 110 areoptionally configured to support more than one game player. For example,a game console may be configured to support two, three, four or moresimultaneous players. Each of these players may receive a separate videostream, or a single video stream may include regions of a framegenerated specifically for each player, e.g., generated based on eachplayer's point of view. Clients 110 are optionally geographicallydispersed. The number of clients included in Game System 100 may varywidely from one or two to thousands, tens of thousands, or more. As usedherein, the term “game player” is used to refer to a person that plays agame and the term “game playing device” is used to refer to a deviceused to play a game.

Clients 110 are configured to receive video streams via Network 115.Network 115 may be any type of communication network including, atelephone network, the Internet, wireless networks, powerline networks,local area networks, wide area networks, private networks, and/or thelike. In typical embodiments, the video streams are communicated viastandard protocols, such as TCP/IP or UDP/IP. Alternatively, the videostreams are communicated via proprietary standards.

A typical example of Clients 110 is a personal computer comprising aprocessor, non-volatile memory, a display, decoding logic, networkcommunication capabilities, and input devices. The decoding logic mayinclude hardware, firmware, and/or software stored on a computerreadable medium. Systems for decoding (and encoding) video streams arewell known in the art and vary depending on the particular encodingscheme used.

Clients 110 may, but are not required to, further include systemsconfigured for modifying received video. For example, a client may beconfigured to perform further rendering, to overlay one video image onanother video image, to crop a video image, and/or the like. Forexample, Clients 110 may be configured to receive various types of videoframes, such as I-frames, P-frames and B-frames, and to process theseframes into images for display to a user. In some embodiments, a memberof Clients 110 is configured to perform further rendering, shading,conversion to 3-, or like operations on the video stream. A member ofClients 110 is optionally configured to receive more than one audio orvideo stream. Input devices of Clients 110 may include, for example, akeyboard, a joystick, a pointing device, a force feedback device, amotion and/or location sensing device, a mouse, a touch screen, a neuralinterface, a camera, and/or the like.

The video stream (and optionally audio stream) received by Clients 110is generated and provided by Video Server System 120. As is describedfurther elsewhere herein, this video stream includes video frames (andthe audio stream includes audio frames). The video frames are configured(e.g., they include pixel information in an appropriate data structure)to contribute meaningfully to the images displayed to the user. Ameaningful contribution is a contribution that is readily observable bya game player. As used herein, the term “video frames” is used to referto frames including predominantly information that is configured tocontribute to, e.g. to effect, the images shown to the user. Most of theteachings herein with regard to “video frames” can also be applied to“audio frames.”

Clients 110 are typically configured to receive inputs from a user.These inputs may include game commands configured to change the state ofthe video game or otherwise affect game play and/or operation of a videogame. The game commands can be received using input devices and/or maybe automatically generated by computing instructions executing onClients 110. The received game commands are communicated from Clients110 via Network 115 to Video Server System 120 and/or Game Server 125.For example, in some embodiments, the game commands are communicated toGame Server 125 via Video Server System 120. In some embodiments,separate copies of the game commands are communicated from Clients 110to Game Server 125 and Video Server System 120. The communication ofgame commands is optionally dependent on the identity of the command.Game commands are optionally communicated from Client 110A through adifferent route or communication channel that that used to provide audioor video streams to Client 110A.

Game Server 125 is optionally operated by a different entity than VideoServer System 120. For example, Game Server 125 may be operated by thepublisher of a multiplayer game. In this example, Video Server System120 is optionally viewed as a client by Game Server 125 and optionallyconfigured to appear from the point of view of Game Server 125 to be aprior art client executing a prior art game engine. Communicationbetween Video Server System 120 and Game Server 125 optionally occursvia Network 115. As such, Game Server 125 can be a prior art multiplayergame server that sends game state information to multiple clients, oneof which is Video Server System 120. Video Server System 120 may beconfigured to communicate with multiple instances of Game Server 125 atthe same time. For example, Video Server System 120 can be configured toprovide a plurality of different video games to different users. Each ofthese different video games may be supported by a different Game Server125 and/or published by different entities. In some embodiments, severalgeographically distributed instances of Video Server System 120 areconfigured to provide game video to a plurality of different users. Eachof these instances of Video Server System 120 may be in communicationwith the same instance of Game Server 125. Communication between VideoServer System 120 and one or more Game Server 125 optionally occurs viaa dedicated communication channel. For example, Video Server System 120may be connected to Game Server 125 via a high bandwidth channel that isdedicated to communication between these two systems.

Video Server System 120 comprises at least a Video Source 130, an I/ODevice 145, a Processor 150, and non-transitory Storage 155. VideoServer System 120 may include one computing device or be distributedamong a plurality of computing devices. These computing devices areoptionally connected via a communications system such as a local areanetwork.

Video Source 130 is typically configured to provide a video stream,e.g., streaming video or a series of video frames that form a movingpicture. In some embodiments, Video Source 130 is also configured toprovide an audio stream. In some embodiments, Video Source 130 includesa video game engine and rendering logic. The video game engine isconfigured to receive game commands from a player and to maintain a copyof the state of the video game based on the received commands. This gamestate includes the position of objects in a game environment, as well astypically a point of view. The game state may also include properties,images, colors and/or textures of objects. The game state is typicallymaintained based on game rules, as well as game commands such as move,turn, attack, set focus to, interact, use, and/or the like. Part of thegame engine is optionally disposed within Game Server 125. Game Server125 may maintain a copy of the state of the game based on game commandsreceived from multiple players using geographically disperse clients. Inthese cases, the game state is provided by Game Server 125 to VideoSource 130, wherein a copy of the game state is stored and rendering isperformed. Game Server 125 may receive game commands directly fromClients 110 via Network 115, and/or may receive game commands via VideoServer System 120.

In alternative embodiments, Video Source 130 is configured to provideimages that do not form a moving picture. These images may be the imageoutput of a game application or, as discussed elsewhere herein, theimage output of a non-game application.

Video Source 130 typically includes rendering logic, e.g., hardware,firmware, and/or software stored on a computer readable medium such asStorage 155. This rendering logic is configured to create video framesof the video stream based on the game state. All or part of therendering logic is optionally disposed within a graphics processing unit(GPU). Rendering logic typically includes processing stages configuredfor determining the three-dimensional spatial relationships betweenobjects and/or for applying appropriate textures, etc., based on thegame state and viewpoint. The rendering logic produces raw video that isthen usually encoded prior to communication to Clients 110. For example,the raw video may be encoded according to an Adobe Flash® standard,.wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x. Xvid.FFmpeg, x264, VP6-8, realvideo, mp3, or the like. The encoding processproduces a video stream that is optionally packaged for delivery to adecoder on a remote device. The video stream is characterized by a framesize and a frame rate. Typical frame sizes include 800×600, 1280×720(e.g., 720p), 1024×768, although any other frame sizes may be used. Theframe rate is the number of video frames per second. A video stream mayinclude different types of video frames. For example, the H.264 standardincludes a “P” frame and an “I” frame. I-frames include information torefresh all macro blocks/pixels on a display device, while P-framesinclude information to refresh a subset thereof. P-frames are typicallysmaller in data size than are I-frames. As used herein the term “framesize” is meant to refer to a number of pixels within a frame. The term“frame data size” is used to refer to a number of bytes required tostore the frame. Rendering logic is optional in embodiments where theoutput of Video Source 130 does not include images generated in from agame environment or the like.

In alternative embodiments Video Source 130 includes a video recordingdevice such as a camera. This camera may be used to generate delayed orlive video that can be included in the video stream of a computer game.The resulting video stream optionally includes both images rendered inreal-time and images recorded using a still or video camera. VideoSource 130 may also include storage devices configured to storepreviously recorded video to be included in a video stream. Video Source130 may also include motion or positioning sensing devices configured todetect motion or position of an object, e.g., person, and logicconfigured to determine a game state or produce video-based on thedetected motion and/or position.

Video Source 130 is optionally configured to provide overlays configuredto be placed on other video. For example, these overlays may include acommand interface, log in instructions, messages to a game player,images of other game players, video feeds of other game players (e.g.,webcam video). In embodiments of Client 110A including a touch screeninterface, the overlay may include a virtual keyboard, joystick, touchpad, and/or the like. In one example of an overlay a player's voice isoverlaid on an audio stream. Video Source 130 optionally furtherincludes one or more audio sources.

As discussed elsewhere herein, overlays are optionally used to cover orotherwise modify elements within a video image generated by a gameengine. For example, an input control that is undesirable when a videogame is executed on a Video Server System 120 (but perhaps notundesirable when executed locally on a personal computer) can be coveredusing an overlay. An input control is a clickable, selectable orotherwise actionable area of an image provided by Video Server System120 that is tied to underlying functionality. The underlyingfunctionality optionally includes generation of an input command, suchas a game command. Not all input commands are generated by an inputcontrol. For example, some input commands are generated usingkeystrokes. Input commands may be configured to operate on the localmember of Clients 110 or be configured to operate on software orhardware that happens to be remote from the local member of Clients 110.For example, an input command may be configured to operate on a gameengine within Video Source 130. In some embodiments, input commandsinclude game commands as well as other commands initiated by a gameplayer.

In embodiments wherein Video Server System 120 is configured to maintainthe game state based on input from more than one player, each player mayhave a different point of view comprising a position and direction ofview. Video Source 130 is optionally configured to provide a separatevideo stream for each player based on their point of view. Further,Video Source 130 may be configured to provide a different frame size,frame data size, and/or encoding to each of Client 110. Video Source 130is optionally configured to provide 3-D video.

I/O Device 145 is configured for Video Server System 120 to send and/orreceive information such as video, commands, requests for information, agame state, client identities, player identities, game commands,security information, audio data, and/or the like. I/O Device 145typically includes communication hardware such as a network card ormodem. I/O Device 145 is configured to communicate with Game Server 125,Network 115, and/or Clients 110. I/O Device 145 is configured to receivethe information from more than one of Clients 110. I/O Device 145 isoptionally configured to receive the information as packets using astandard such as TCP or UDP.

Processor 150 is configured to execute logic, e.g. software, includedwithin the various components of Video Server System 120 discussedherein. For example, Processor 150 may be programmed with softwareinstructions in order to perform the functions of Video Source 130, GameServer 125, and/or a Client Qualifier 160. Video Server System 120optionally includes more than one instance of Processor 150. Processor150 may also be programmed with software instructions in order toexecute commands received by Video Server System 120, or to coordinatethe operation of the various elements of Game System 100 discussedherein. Processor 150 may include one or more hardware device. Processor150 is an electronic or optical processor.

Storage 155 includes non-transitory analog and/or digital storagedevices. For example, Storage 155 may include an analog storage deviceconfigured to store video frames. Storage 155 may include a computerreadable digital storage, e.g. a hard drive, an optical drive, or solidstate storage. Storage 115 is configured (e.g. by way of an appropriatedata structure or file system) to store video frames, artificial frames,a video stream including both video frames and artificial frames, audioframe, an audio stream, overlays, command mappings, lists of blockedcommands, bitmaps, and/or the like. Storage 155 is optionallydistributed among a plurality of devices. In some embodiments, Storage155 is configured to store the software components of Video Source 130discussed elsewhere herein. These components may be stored in a formatready to be provisioned when needed.

Video Server System 120 optionally further comprises Client Qualifier160. Client Qualifier 160 is configured for remotely determining thecapabilities of a client, such as Clients 110A or 110B. Thesecapabilities can include both the capabilities of Client 110A itself aswell as the capabilities of one or more communication channels betweenClient 110A and Video Server System 120. For example, Client Qualifier160 may be configured to test a communication channel through Network115.

Client Qualifier 160 can determine (e.g., discover) the capabilities ofClient 110A manually or automatically. Manual determination includescommunicating with a user of Client 110A and asking the user to providecapabilities. For example, in some embodiments, Client Qualifier 160 isconfigured to display images, text, and/or the like within a browser ofClient 110A. The displayed objects request that the user enterinformation such as operating system, processor, video decoder type,type of network connection, display resolution, etc. of Client 110A. Theinformation entered by the user is communicated back to Client Qualifier160.

Automatic determination may occur, for example, by execution of an agenton Client 110A and/or by sending test video to Client 11˜A. The agentmay comprise computing instructions, such as java script, embedded in aweb page or installed as an add-on. The agent is optionally provided byClient Qualifier 160. In various embodiments, the agent can find outprocessing power of Client 110A, decoding and display capabilities ofClient 110A, lag time reliability and bandwidth of communicationchannels between Client 110A and Video Server System 120, a display typeof Client 110A, firewalls present on Client 110A, hardware of Client110A, software executing on Client 110A, registry entries within Client110A, and/or the like.

Client Qualifier 160 includes hardware, firmware, and/or software storedon a computer readable medium. Client Qualifier 160 is optionallydisposed on a computing device separate from one or more other elementsof Video Server System 120. For example, in some embodiments, ClientQualifier 160 is configured to determine the characteristics ofcommunication channels between Clients 110 and more than one instance ofVideo Server System 120. In these embodiments the information discoveredby Client Qualifier can be used to determine which instance of VideoServer System 120 is best suited for delivery of streaming video to oneof Clients 110.

FIG. 2 is a block diagram of embodiments of Video Source 130 configuredto serve multiple video games. The multiple video games can includemultiple instances of the same video game and/or instances of differentvideo games. The video games are optionally multiplayer games. Forexample, a game state of one or more of the video games may bemaintained by one or more instances of Game Server 125 each based oninputs received from multiple clients. The elements of Video Source 130illustrated in FIG. 2 are optionally executed using Processor 150.

The video games are executed using different instances of Game Logic210, individually labeled 210A, 210B, 210C, etc. Game Logic 210 isconfigured to use a game state to determine a game environment that canbe rendered to a video stream configured to be presented to a gameplayer. The game environment is a three dimensional virtual environmentincluding in-game objects, avatars, locations of objects, their shapes,textures, and spatial relationships there between, and the like. A gameenvironment can include vertex data, transformation data and texturedata, and/or the like. In alternative embodiments one or more of GameLogic 210 are replaced by other application logic. In these embodimentsthe output of Video Source 130 can include video, single images, and/ormultiple images at a variable frame rate. Frame rate may vary, forexample, as a function of how often an image is updated by anapplication. In some embodiments images are provided in a video formatbut new frames are typically only provided when the generated imagechanges.

The rendering of the game environment is typically based on one or morepoint of view associated with a specific game player. Video Source 130may include more than 1, 2, 3, 5, 7, 15, or 31 instances of Game Logic210. Game Logic 210 is optionally a client of Game Server 125 and may beconfigured to communicate with Game Server 125 via Network 115.

Game Logic 210 is configured to receive game commands from one or moreof Clients 110 and to process the received commands according to a setof game rules. These rules cover, for example, how avatars interact withother avatars or in game objects, avatar movement, game instancemanagement, and/or the like. Game Logic 210 is optionally alsoconfigured to generate audio data based on events within the game. Thisaudio data may represent a gunshot, a splash, an engine, voice, flying,rain, music, or any other sound that could occur in a game. For example,an event such as one object hitting another may result in audio datarepresenting a related sound. Game Logic 210 includes hardware,firmware, and/or software stored on a computer readable medium.

Each instance of Game Logic 210 can be disposed on a separate computingdevice or several instances of Game Logic 210 could be disposed on thesame computing device or a single instance of Game Logic 210 could bedisposed across multiple computing devices. Game instances candynamically change the number and/or identify of computing devices useddepending on the requirements of the game environment the user iscurrently experiencing. Instances of Game Logic 210 disposed on the samecomputing device are optionally executed within separate virtualmachines or virtual 110 shells. In some embodiments, different instancesof Game Logic 210 are configured to communicate game commands and/orgame state information directly to each other, e.g., without necessarilycommunication through Game Server 125.

Game Logic 210 typically executes on top of an Operating System (OS)215. Operating System 215 may include Windows™, Linux, UNIX, Mac OS™,Solaris™, and/or the like may operate between Operating System 215 andGame Logics 210. Virtual machine platforms such as ESX, Hyper-V, and/orthe like. In these embodiments, one or more of Game Logic 210 can beexecuted within a virtual machine. Multiple instances of Game Logic 210may execute on the same instance of Operating System 215. For example,FIG. 2 illustrates Game Logic 210A and Game Logic 210B both executing onthe same Operating System 215. Instances of Game Logic 210 executing onthe same Operation System 215 may, or may not, be configured for playingthe same video game. For example, Game Logic 210A, 210B and 210C may allbe World of Warcraft® clients, or may be clients of World of Warcraft®,Eve® and Call to Arms® respectively.

In some embodiments, the game environments determined by Game Logic 210are passed to an optional Virtual 3D Video Driver 220. Virtual 3D VideoDriver 220 is configured to appear, from the point of view of Game Logic210, as a non-virtual 3D video driver controlling a graphics processingunit. Each instance of Game Logic 210 may be associated with its owninstance of Virtual 3D Video Driver 220, or Virtual 3D Video Driver 220may be shared by two or more instances of Game Logic 210. In someembodiments, game audio determined by each Game Logic 210 is passed toan optional Virtual Audio Driver 285. Game audio includes sound dataconfigured to be presented to a game player as part of an audio stream.The game audio can result from rules of the game, e.g., shooting a gunshould result in a popping sound, or may be received from other sourcessuch as music tracks, members of Clients 110, a microphone, and/or thelike.

In alternative embodiments, Game Logic 210 is configured to include thefunctionality, as discussed herein, of Virtual 3D Video Driver 220and/or Virtual Audio Driver 285. In these embodiments, Virtual 3D VideoDriver 220 and/or Virtual Audio Driver 285 are not required.

Virtual 3D Video Driver 220 is further configured to pass the receivedgame environments to a (non-virtual) 3D Driver 230. Optionally thedelivery of game environments to 3D Driver 230 is coordinated by thevarious instances of Virtual 3D Video Driver 220. For example, deliverycan be coordinated such that 3D Driver 230 receives game environmentsfrom only one or a minimum number of Virtual 3D Video Driver 220 at atime. In typical embodiments, each of Virtual 3D Video Drivers 220 isconfigured such that they appear to be a separate process and a separatesource of video data to 3D Driver 230. As such, 3D Driver 230 isconfigured to keep track of which video data results in which videoframes after rendering.

The video data received by 3D Driver 230 are passed to GraphicsProcessing Unit (GPU) 235 for rendering into raw video frames. GraphicsProcessing Unit 235 is optionally used to render more than one videostream in parallel. For example, Graphics Processing Unit 235 maygenerate a first video stream based on video data received from GameLogic 210A, generate a second video stream based on video data receivedfrom Game Logic 210B, generate a third video stream based on video datareceived from Game Logic 210C, etc. in parallel. The parallel productionof video streams includes the generation of these streams at the sametime. However, parallel production of video streams may, but does notnecessarily, include the processing of individual frames at the sametime within Graphics Processing Unit 235. For example, in someembodiments 3D Driver 230 alternatively passes the video data generatedby the various members Game Logic 210 to Graphics Processing Unit 235.Data generated by Game Logic 210A is used to make a video frame, andsubsequently data generated by Game Logic 2108 is used to make a videoframe, etc. In this case, the video streams are produced in parallelwhile individual frames are produced in series. Alternatively, more thanone video frame may be generated within Graphics Processing Unit 235 atthe same time. In this case a first part of Graphics Processing Unit 235is used to generate one frame while a second part of Graphics ProcessingUnit 235 is used to generate a second frame, the first and second framesarising from video data produced by different Game Logic 210. In someembodiments, one set of graphical computation cores is used to generatethe first frame while a second set of graphical computation cores isused to generate the second frame, at the same time. The resulting videoframes are passed back to the control of 3D Driver 230.

Virtual 3D Video Drivers 220 are typically configured to manage thetransfer of raw rendered video frames from 3D Driver 230. For example,Virtual 3D Video Drivers 220 may be used to coordinate the transfer ofvideo frames from the control of 3D Driver 230 to a Shared Memory 240.(Following rendering, the video frames are in a memory of GraphicsProcessing Unit 235 or a memory managed by 3D Driver 230. In either casethey are under the control of 3D Driver 230.) As Virtual 3D VideoDrivers 220 manages the communication of video data and frames to andfrom 3D Driver 230, in some embodiments, they are capable of placing thevideo frames within parts of Shared Memory 240 associated with specificVideo Dynamic Encoding Pipelines (DEPs) 245. The Video DEPs 245 areindividually identified as Video DEP 245A, Video DEP 2458, Video DEP245C, etc. In these embodiments, each Video DEP 245 is assigned specificmemory locations within Shared Memory 240, and is configured to retrievevideo data from those locations.

In other embodiments, Virtual 3D Video Drivers 220 are configured tomanage the transfer of video data to Shared Memory 240 based on timing.In these embodiments, the transfers managed by each of Virtual 3D VideoDrivers 220 are synchronized and the Virtual 3D Video Drivers 220 notifyeach Video DEP 245 when data for them is in Shared Memory 240. Once thisdata is retrieved by the notified member of Video DEPs 245, data foranother member of DEPs is transferred from 3D Driver 230 to SharedMemory 240 under the control of Virtual 3D Video Drivers 220. In thesecases pixel read back from the local GPU memory to Shared Memory 240 canbe based on a schedule. As used herein the terms “in sync” and“synchronized” are meant to mean that the two events are related in timeby a schedule, by a timing signal, by a time delay, or that one eventdoes not start until the other event has satisfied a condition, e.g.,has completed. For example, Virtual 3D Video Drivers 220 may operate insync such that a game environment is provided to 3D Driver 230 from afirst of Virtual 3D Video Drivers 220 once Graphics Processing Unit 235has completed rendering of a game environment from a second of Virtual3D Video Drivers 220. The timing of this synchronization is optionallyselected to make optimum use of Graphics Processing Unit 235.

As a result of the management by Virtual 3D Video Drivers 220, themultiple video streams can be stored in Shared Memory 240 withoutconfusing which frames belong to which video stream. Shared Memory 240is optionally configured to store audio as well as video data. Thisaudio data may be stored and/or managed in ways similar to thatdiscussed with respect to video data.

Virtual Audio Drivers 285 are optionally part of a virtual I/0 shellbetween Game Logic 210 and OS 215. They are configured to appear fromthe point of view of Game Logic 210 as an audio driver and to pass anyaudio data they receive from Game Logic 210 to one of Audio DEP 270. Forexample, in some embodiments, the Virtual Audio Driver 285 associatedwith Game Logic 210 is configured to send audio data to Audio DEP 270A,and the Virtual Audio Driver 285 associated with Game Logic 210B isconfigured to send audio data to Audio DEP 270B, etc.

Shared Memory 240 includes random access memory (RAM) or a similarmemory configured for efficient reading and writing of video data.Shared Memory 240 is configured to store video data for a plurality ofdifferent Video DEP 245. Video data for different Video DEPs 245 isoptionally stored at the same time in Shared Memory 240. Shared Memory240 may consist of a single hardware device or may include multipledevices.

Video DEPs 245 are dynamically allocated encoding pipelines that areeach configured to encode video data rendered using Graphics ProcessingUnit 235. Each member of Video DEPs 245 is configured to encode to videoformats specified at the time the Video DEP 245 is provisioned. Thisformat specification is typically based on the needs of one of Clients110 and/or the capabilities of the communication path between VideoServer System 120 and the Client 110. Video DEPs 245 are optionallyprovisioned dynamically in response from a request from one of Clients110. For example, when Client 110A connects to Video Server System 120and sends a request for a video stream, Video DEP 245 can be provisionedto include elements, such as an encoder, selected specifically to meetneeds of Client 110A. As is discussed elsewhere herein, a member ofVideo DEPs 245 is optionally configured to use more than one encodingscheme.

Video DEPs 245 each include a Grabber 247 configured to grab video datafrom Shared Memory 240 and transfer this video data to a memory of theVideo DEP 245. The Grabber 247 is optionally under the control of amember of Virtual 3D Video Drivers 220. For example, in variousembodiments, Grabber 247 is controlled by a member of Virtual 3D VideoDrivers 220 to grab the video data from a specific location withinShared Memory 240 and/or at a specific time in synchronization with thetransfer of video data to other members of Video DEPs 245. In someembodiments, Grabber 247 is also configured to grab 2D video datadirectly from Memory XII 225.

Video DEP 245 each optionally include an Overlay Logic 290 configured tooverlay one or more video images on the video data, e.g., frames,generated by Graphics Processing Unit 235. An overlay is a first imageto be placed on a second image, or a sound added to another sound.Overlays may be applied with varying levels of transparency. Forexample, the first image can be opaque and, thus, completely hidecovered parts of the second image, or the first image can be partiallytransparent in which case the second image can still be somewhat visibleunder the first image. An overlay may cover all or part(s) of theunderlying image. For example, an overlay may be configured to covercertain pixels within a larger area. In this case the overlay is mappedto pixels on a display of one of Clients 110 or mapped to pixels withinan image generated by Graphics Processing Unit 235. The mapping can beaccomplished in various ways. For example, a lower left pixel of theoverlay may be assigned to a specific pixel on a video frame generatedby Graphics Processing Unit 235. In some embodiments, Overlay Logic 290is configured to resize an overlay prior to applying the overlay.Overlay Logic 290 is optionally located in other positions within VideoDEP 245A. For example, after Preprocessor 250, Overlay Logic 290 isoptionally configured to apply more than one overlay. In someembodiments, Video DEP 245 includes a plurality of Overlay Logic 290configured to apply overlays to the same or different frames. A secondoverlay can be placed on top of a first overlay.

The overlays can include a wide variety of pixel/video images. Forexample, an overlay can include a real-time or static image of a gameplayer received via the internet, a real-time or static image of asporting (or other) event, an image of a input control (e.g., a button,a link, a virtual joystick, a slide bar, a wheel, a clickable object,“a,” “b,” “TAB,” “Return,” “Space Bar,” “Function Key,” “Arrow,” and/orother keys or input devices), a map, text, and/or the like. Overlays canvary widely in size and shape. In some cases the overlay is the resultof rendering a 3D game environment. More than one overlay can be placedon an image. These overlays can be overlapping or separate. Overlays areoptionally received in an encoded format and decoded prior to beingapplied. Overlays are optionally stored in Storage 155.

The presence and content of overlays are optionally responsive toreceived game commands, identity of one or more game players, an accounttype, identity of a game being played, video captured from a prior orreal-time sporting event or real-world game, game rules, and/or thecontent of video generated by Game Logic 210. For example, an overlaymay include a menu resulting from receipt of a game command. An overlaymay include content responsive to a pixel pattern, e.g., image, includedwithin video generated by Game Logic 210. Overlay Logic 290 isoptionally configured to apply multiple overlays to a single video frameand these overlays may be from different sources.

Overlay Logic 290 is optionally configured to apply an overlay after avideo frame has been preprocessed using Preprocessor 250 and/or afterencoding using one of Encoders 255. In some embodiments the overlayincludes an image received at an input of Video Source 130 (e.g., I/ODevice 145) via the internet and Overlay Logic 290 includes videoprocessing logic configured to generate the overlay from the image. Thereceived image is optionally received from an image source discussedelsewhere herein, e.g., and Image Source 630 discussed in reference toFIG. 6.

Video DEPs 245 each optionally also include a Preprocessor (PRE) 250.Preprocessor 250 is configured to perform a color space conversion suchas RGB to YUV and/or a scaling operation to increase or decrease theresolution of the video frame. Preprocessor 250 is optional inembodiments wherein the output of Graphics Processing Unit 235 is in theYUV color space or some other desired color space. Multiple Preprocessor250 may be included in a Video DEP 245 configured to produce multiplevideo streams having video frames of different sizes.

Video DEPs 245 each include at least one Encoder (ENC) 255. Encoders 255are individually identified as Encoder 255A, Encoder 255B, etc. Each ofEncoders 255 is configured to encode the video data according to aspecific codec, and optionally a specific color depth and/or frame size.For example, Encoders 255 may be configured to encode video data (orstill images) to, a custom format, Adobe Flash® standard, .flv, .wav,.avi, .mpg, H.264, H.263, On2, VP6, VC-1, TIFF, JPEG, WMF, GIF, and/orother codecs discussed herein.

A member of Video DEPs 245 may include one, two or more Encoders 255.These encoders may be configured to encode to different codecs and/orthe different formats of the same codec. For example, Encoder 255A maybe configured to encode according to a Flash standard at a first framesize and color depth while Encoder 255 is configured to encode to thesame Flash standard at a second frame size and color depth. The identityof Encoders 255 within each member of Video DEPs 245 is typicallydetermined at the time the Video DEP 245 is provisioned. For example, asingle command or command set may be used to create (provision) VideoDEP 245A and specify which components the created Video DEP 245A shouldinclude. The creation of Video DEP 245A is discussed further elsewhereherein. A member of Video DEPs 245 including two or more Encoders 255 isalternatively viewed as two or more separate Video DEPs 245 having somebut not all components in common. For example, Video DEP 245A and VideoDEP 2458 may have the same Preprocessor 250 but different Encoders 255.

In one example, Encoder 255A is configured to use a codec for H.264while Encoder 2558 is configured to use a codec for H.263. Having two ormore different encoders available enables Video DEP 245A to changeencoding during delivery of a video stream. The change in encoding canbe from one type of encoding to another, or merely a change incharacteristics of a specific type of coding. For example, thecharacteristics may change in terms of color depth, number of frames persecond, encoding options, number of pixels, and/or the like. In someembodiments, Video DEP 245A is optionally configured to switch betweenEncoder 255A and 255B in response to a change in the characteristics ofClient 110A or the communication channel between Client 110A and VideoSource 130.

In practice, when a different codec other than that being used isrequired a new Video DEP 245 is spawned and executed in parallel withthe current Video DEP 245 for a short period of time. The new Video DEP245 is optionally a branch of the original Video DEP 245. For example,some components of the original Video DEP 245 may be used in the newVideo DEP 245. These two Video DEP 245 may have components logicallyarranged in a tree structure.

In some embodiments, Video DEP 245A is configured to use two or moredifferent encoders to generate two or more different video streams atthe same time. These video streams are based on the same gameenvironment rendered by Graphics Processing Unit 235 and, thus, includeessentially the same materials (with the possible exception of overlays)and can be sent to different places. For example, one of the videostreams can be sent to Client 110A while the other is sent to Client110B. Alternatively, one of the video streams can be sent to Client 110Aand the other to a website where third parties can watch the video. Thiswebsite is optionally part of a social networking site or a gameplayer's site. The two different video streams may be different in theframe rate, encoding type, frame size, color depth, etc. For example avideo stream delivered to a social networking website can be of muchlower quality than the video stream delivered to a game player who isplaying a game using Client 110A. The second video stream may bedirected to a game player who is playing the game or to people who aremerely observing the game play. A video stream is optionally directed tomore than one place.

Video DEP 245A optionally includes one or more Post Processors (Post)260. Individual examples of Post Processors 260 are labeled 260A and260B. Post Processors 260 are configured to package an encoded videoframe in a container in a format appropriate for communication over anetwork according to a public or proprietary network protocol. Forexample, some protocols such as Adobe RTMP require post processing whileother video standards such as H.264 Annex B do not require postprocessing. Each of Post Processors 260 may be associated with aspecific member of Encoders 255, or several Post Processors 260 may beconfigured to receive encoded video frames from one member of Encoders255.

The output of Post Processors 260 is directed to I/O Device 145 fordelivery to one or more of Clients 110. The Elements of Video DEPs 245discussed herein include hardware, firmware and/or software stored on acomputer readable medium. For example, each of Video DEPs 245 mayrepresent a set of software loaded into memory and executing using anelectronic processor.

The operation of Video Source 130 is typically managed by a Controller265. Controller 265 includes hardware, firmware and/or software storedon a computer readable medium. For example Controller 265 may includesoftware stored in memory and executed using a microprocessor.

Audio DEPs 270 are configured to receive audio data generated by membersof Game Logic 210 and encode this audio data to audio packets. Theencoded data packets are then sent to Clients 110 using I/O Device 145and Network 115. Audio DEPs 270 are also optionally configured to placean audio overlay on the audio data received from Game Logic 210.Different Audio DEPs 270 are individually labeled 270A, 270B, 270C, etc.

As with Video DEPs 245, Audio DEPs 270 are typically allocated as neededand assigned to process audio data from a specific member of Game Logic210. This audio data may be based on events within a video game. Forexample, actions taken by one or more players of the video game mayresult in audio data according to game rules (e.g., falling in watermakes a splashing sound). Allocation of Audio DEP 270A is typicallyresponsive to the same factors and performed in the same manner asallocation of Video DEP 245A. For example, the elements included inAudio DEP 270A may be responsive to needs of one of Clients 110. AudioDEP 270 may receive audio data from Shared Memory 240 and/or directlyfrom one of Virtual Audio Drivers 285.

Audio DEP 270A optionally includes a Grabber 275 configured to receiveaudio data from one of Virtual Audio Drivers 285, to receive audio datafrom one or more sources external to Video Server System 120, and/or toget audio data from Shared Memory 240. For example, in some embodimentsGrabber 275 is configured to retrieve data received from sourcesexternal to Video Source 130, such as one or more of Clients 110, overthe internet. The audio received, from any of these sources, can includevoice audio, music, sound effects, sound tracks, alerts, and/or thelike. For example, the voice of a game player using Client 110A may bereceived from Client 110A, or the game player may provide music to beheard by a game player using another of Clients 110. Audio data may bereceived, as audio streams, from more than one game player in parallel.

Audio DEP 270A optionally further includes Overlay Logic 295. OverlayLogic 295 is configured to place one or more audio overlays on audiodata generated by a member of Game Logic 210. The audio overlays caninclude audio, music, sound effects, sound tracks, alerts, and/or thelike. For example, in some embodiments, an audio overlay includes voicedata received as data packets from one or more game players over theinternet. These data packets are optionally decoded by Overlay Logic 295prior to being included in the overlay. Placement of overlays on theaudio data generated by a member of Game Logic 210 optionally includesscaling of the audio data within the overlay and addition of this datato the generated audio data. All or part of an audio overlay can bereceived from a source external to Audio Source 130. For example, insome embodiments the overlay includes audio data received at I/O Device145 via the internet and Overlay Logic 290 includes audio processinglogic configured to generate an audio overlay from this audio data. Whenreferring to audio overlays, the identities of which data is the overlayand which is the data on which the overlay is being applied may bearbitrary as, in some embodiments, audio data does not have a sense oforder that can be possible in video data.

Audio DEP 270A further includes an Audio Encoder 280A. Audio Encoder280A is configured to encode the audio data generated by a member ofGame Logic 210 and the audio overlay into an audio packet according toan audio codec. The encoding is typically performed after the audiooverlay(s) have been applied to the audio data.

Grabber 275, Audio Encoder 280A and Overlay Logic 295 include hardware,firmware, and/or software stored on a computer readable medium. AudioDEP 270B and Audio DEP 270C are alternative embodiments of Audio DEP270A.

In some embodiments, Controller 265 is configured to provision instancesof Game Logic 210A, Virtual Audio Driver 285 and Virtual 3D Video Driver220 in response to a request to play a game. For example, if a requestfor a specific video game is received from Client 110A, Controller 265may retrieve Game Logic 210A, Virtual Audio Driver 285 and Virtual 3DVideo Driver 220 from Storage 155 and place these in working memory.Game Logic 210A may be placed in communication with both Client 110A andGame Server 125. Game Logic 210A is then executed using Processor 150 inorder to play the video game.

In some embodiments, Controller 265 is configured to allocate instancesof Video DEP 245 and/or Virtual Audio Driver 285 in response to arequest to play a game. For example, Controller 265 may be configured tofirst determine or receive the requirements for Video DEP 245A and AudioDEP 270A and then provision computing instructions that satisfy thoserequirements. Video DEP 245A and Audio DEP 270A are then associated withone of Game Logic 210 and one of Virtual 3D Driver 220. The provisionedVideo DEP 245A is optionally configured to retrieve raw video data froma specific area within Shared Memory 240.

In some embodiments the requirements for Video DEP 245A and/or Audio DEP270A are determined by querying a user of Client 110A. For example,Controller 265 may be configured to cause text messages to appear onClient 110A, the messages requesting that a user of Client 110A entercharacteristics of a video stream desired by a user. Thesecharacteristics can include a connection type, a frame size, an encodingscheme, a frame rate, a color depth, and/or the like. The enteredcharacteristics are conveyed back to Controller 265, which then usedthese characteristics to select elements to include in Video DEP 245Aand/or Audio DEP 270A when they are provisioned. For example, if theuser requests a specific encoding scheme, then an Encoder 255 configuredto use the requested encoding scheme is selected and included in VideoDEP 245A.

In some embodiments the requirements for Video DEP 245A and/or Audio DEP270A are determined automatically. For example, information gatheredusing Client Qualifier 160 in an Execute Agent Step 310 (See FIG. 3) canbe used to determine the requirements for Video DEP 245A. In thisexample, if Client 110A includes only a Microsoft Silverlight® decoderand an Adobe Flash® decoder, then at least one of the correspondingencoder configurations or variants thereof is required in Video DEP245A. Characteristics of the video stream to be generated using VideoDEP 245A, such as frame size, color depth and frame rate, can also bedetermined by Controller 265 in this way.

The determination of requirements for Video DEP 245A and/or Audio DEP270 may include both the use of characteristics provided by a user andcharacteristics automatically determined. In some embodiments, thepossible characteristics are limited by a user's account type, paymentand/or subscription. For example, a user may pay to receive a higherquality video stream.

Controller 265 may also manage sharing of Shared Memory 240. Forexample, Controller 265 may configure Video DEP 245A and/or Audio DEP270A to use specific memory locations within Shared Memory 240 and/orController 265 may be configured to configured Video DEP 245A and/orAudio DEP 270A to access Shared Memory 240 in response to certain timingsignals. The timing signals being timed for synchronization of access toShared Memory 240 by different Video DEPs 245.

FIG. 3 illustrates methods of receiving a qualified request for a videostream from a client such as Client 110A, according to variousembodiments of the invention. A qualified request, as the term is usedherein, is a request that is received from a client, where the clienthas been qualified to receive the video stream. The qualification means,for example, that the client meets a minimum level of decodingcapability and/or that the communication channel between the client andVideo Server System 120 meets minimum bandwidth and/or reliabilityrequirements. In alternative embodiments, the request for a video streamis received from an unqualified client and the client is then qualifiedbefore the video stream is provided to the client.

In a Receive Request Step 300, a request is received to display anobject on Client 110A. This object includes HTML, XML, text, an image,and/or the like, and is configured to indicate that a video stream isavailable for viewing on Client 110A. For example, the object may be abanner advertisement or some other type of link that indicates that auser can play a video game using Client 110A byclicking/selecting/cursor over the object. The request may be receivedas the result of the user loading a web page including a reference tothe object. This reference can include a universal resource locator,HTML, XML, and/or the like. For example, in some embodiments a firstuser chooses to include a link and image of a video game they play ontheir web page within a social networking website. When a second uservisits this page information about the video game is displayed to thesecond user. As a result of computing instructions included within theweb page, Video Server System 120 or some other part of Game System 100receives a request to display text (an object) indicating that thesecond user can play the video game in a streaming mode by clicking onthe object within the displayed information.

The identity of the Video Server System 120 or Game System 100 thatreceives the request in Receive Request Step 300 is optionally dependenton the nature of the request. For example, a request may include a URLthat references a particular game publisher and this request is directedto an embodiment of Game System 100 controlled or licensed to this gamepublisher. The request may come from a specific web page and theidentity of that web page may be used to determine which entity receivesthe request.

In a Provide Agent Step 305, a software agent is provided to Client 110Afrom Video Server System 120. The software agent is configured fordiscovering capabilities of Client 110A and/or capabilities of thecommunication channel between Client 110A and Video Server System 120.The software agent is configured to be executed on Client 110A using anelectronic processor. For example, the software agent may be configuredto examine a registry on Client 110A, identify video decoders installedon Client 110A, identify characteristics of a display of Client 110A,identify an electronic processor of Client 110A, identify firewallsinstalled on Client 110A, identify other software or hardware on Client110A, and/or the like. In some embodiments Client 110A is configured totest the communication channel between Client 110A and Video ServerSystem 120 by sending and receiving data between Client 110A and VideoServer System 120. This data may include test video frames. In someembodiments more than one path between Client 110A and Video ServerSystem 120 is tested. The characteristics of the communication channelstested may include, ping time, lag, jitter, bandwidth, reliability,and/or the like.

In some embodiments the software agent is configured to testcommunication channels between Client 110A and several differentinstances of Video Server System 120. In these embodiments, the softwareagent is capable of determining which of the communication channels ismost capable of delivering a video stream to Client 110A from a memberof the instances of Video Server System 120. As such, the software agentcan determine which instance of Video Server System 120 is preferred forsupplying the video stream.

In some embodiments the software agent includes computing instructionsembedded in a web page. In some embodiments the software agent providedin Provide Agent Step 305 includes a browser plug-in. Provide Agent Step305 is optional in embodiments where the software agent is alreadyinstalled on Client 110A. In some embodiments, the software agent is astandalone program.

In an Execute Agent Step 310 the software agent provided in ProvideAgent Step 305 is executed. The execution of the software agent may beautomatic on delivery of the software agent to Client 110A, or the userloading the web page. The execution of the software is optionallymanually initiated by the user. The software agent is executed using anelectronic processor of Client 110A. The execution of the software agentis optionally initiated by a browser. The execution includes testing ofcommunication channels and/or discovery of client capabilities asdiscussed herein.

In a Receive Capabilities Step 315 the information discovered by thesoftware agent is received by Client Qualifier 160. The informationtypically includes the capabilities and features of Client 110A as wellas characteristics of one or more communication channel between Client110A and one or more Video Server System 120. For example, after theexecution of the software agent on Client 110A the information collectedby the agent may be communicated to I/O Device 145 via Network 115.

In an Approve Client Step 320, Client 110A is approved by ClientQualifier 160 for the purpose of receiving a video stream from VideoSource 130. Approve Client Step 320 includes comparing the informationreceived in Receive Capabilities Step 315 with a set of minimalrequirements for the delivery of streaming video to Client 110A. Theserequirements can include minimum capabilities of Client 110A and/orminimum requirements for the connection between Client 110A and VideoServer System 120. The approval is made only if the minimum(s) are metor acceded. If the minimum(s) are not met or acceded then Client 110A isnot approved for receiving the video stream from Video Source 130. Insome embodiments the minimal requirements for delivery of streamingvideo are selected so as to assure that the video is presented to a userwith at least a desired level of quality. The minimal requirements areoptionally dependent on the identity of the video game associated withthe request or a publisher thereof.

Approve Client Step 320 optionally includes identifying which of aplurality of alternative instances of Video Server Systems 120 is bestsuited to serve the streaming video to Client 110A. For example, theinstance with the best communication channel between Client 110A and theVideo Server Systems 120 may be selected. Criteria for determining whichof Video Server Systems 120 is best suited to serve the streaming videoare discussed elsewhere herein.

In some embodiments Client Qualifier 160 is configured to correctdeficiencies in the capabilities of Client 110A. For example, ClientQualifier 160 can be configured to suggest to a user of Client 110A thata specific decoder be installed, or configured to automatically installa decoder. Client Qualifier 160 can also be configured to suggest to theuser that hardware components of Client 110A should be upgraded.

Once a Client 110A is approved the approval may be stored, for example,on Client 110A or at Client Qualifier 160. If the approval is stored,then Client 110A is optionally preapproved the next time Receive RequestStep 300 occurs and Steps 305-320 are optional. When Receive RequestStep 300 occurs again, the stored approval is retrieved and used todetermine that Client 110A has already been approved. The delivery ofdifferent video streams may require different capabilities of Client110A and/or of the connection between Client 110A and Video Source 130.Therefore, a stored approval optionally includes informationcharacterizing the type of video stream approved and/or the capabilitiesreceived in Receive Capabilities Step 315. In some embodiments, storedinformation is used to qualify the capabilities of Client 110A, but theconnection between Client 110A and Video Server System 120 is retested.

In a Display Object Step 325, the object requested in Receive RequestStep 300 is displayed to the user of Client 110A. In some embodiments,the object is only displayed after Client 110A has been successfullyapproved in Approve Client Step 320. In other embodiments the object isdisplayed but not fully functional until Client 110A has beensuccessfully approved in Approve Client Step 320.

For example, in some embodiments, the object requested in ReceiveRequest Step 300 includes a banner advertisement. The request is made bycode embedded in a web page. When the web page is loaded in a browser ofClient 110A a request is sent to Client Qualifier 160 and ClientQualifier 160 responds by performing Steps 305-320 as needed. Followingapproval of Client 110A the banner advertisement is delivered to Client110A and displayed on the web page in Display Object Step 325.

Alternatively, the banner advertisement is displayed on the web pageprior to Approve Client Step 320, however, the functionality to whichthe banner advertisement is linked to is not fully activated. E.g.,unlike a qualified banner advertisement, clicking on the banneradvertisement will not result in delivery of a video stream. Delivery ofthe video stream is still only performed after Approve Client Step 320is successfully completed.

In alternative embodiments Display Object Step 325 is performed prior toExecute Agent Step 310 and without Approve Client Step 320. In theseembodiments, while the object is displayed it may not be fullyfunctional and/or the actual delivery of the requested video stream isdelayed until after Client 110A is approved in Approve Client Step 320.Thus, the video stream is still not provided to Client 110A until Client110A is qualified. This assures that the video stream will be seen by auser of Client 110A at at least a minimum level of quality.

In a Receive Video Request Step 330, a request for streaming video isreceived by Video Server System 120 from Client 110A. This request andthe streaming video are optionally communicated via Network 115. In someembodiments, the request is received as a result of a user clicking on,cursor over or selecting the object displayed in Display Object Step325. In typical embodiments the streaming video is part of a video gameplayed by the user of Client 110A. As such, the content of the streamingvideo is responsive to game rules and game commands entered by the user,and optionally users of other Clients 110. The request for streamingvideo may be in the form of a request by the user to playa video game ina streaming mode.

In some embodiments, Video Server System 120 is configured such that therequest received in Receive Video Request Step 330 causes game and videorelated resources to be allocated within Video Source 130. FIG. 4illustrates methods of allocating these resources within a Video Source130, according to various embodiments of the invention. These methodsinclude provisioning of Game Logic 210A and Virtual 3D Video Driver 220,and/or dynamic provisioning of Video DEP 245A. The methods illustratedby FIG. 4 are optionally performed using a system manager and a codesource discussed elsewhere herein.

In a Determine Needs Step 410, the needs of Client 110A are determined.These needs may include a desired frame rate, frame size, color depth,encoding, minimum lag time, other needs discussed herein, and/or thelike. The needs of Client 110A may be dependent on a particular videogame played, for example some video games require a lower lag timebetween the time a game player makes a game command and the video thegame player sees reflects the results of the game command, relative toother video games.

In some embodiments, the needs of Client 110A are dependent onselections made by a game player. For example, a player may select aspecific frame size or set the size of a browser window that a frameshould fill. In some embodiments, the needs of Client 110A are selectedto simulate video that would be produced by specific hardware. Forexample, a game player may select to see video as would be produced by aspecific nVidia or a specific AMD graphics card. In this case, the needsare typically specified to reflect the maximum performance of theselected device.

In some embodiments the needs of Client 110A are determined by accessingdata received in Receive Capabilities Step 315 (FIG. 3) and optionallystored in Storage 155. This data is described elsewhere herein. Theneeds of Client 110A are then selected so as to match the capabilitiesof Client 110A. For example, if Client 110A includes an Adobe Flashdecoder then one of the needs may be a video stream encoded in AdobeFlash. If communication to Client 110A has a limited bandwidth, one ofthe needs may be a limit on the frame size that can be transmitted toClient 110A.

In some embodiments, Determine Needs Step 410 includes providing anagent to Client 110A, executing that agent and receiving capabilities ofClient 110A. This approach may include performing embodiments of ProvideAgent Step 305, Execute Agent Step 310 and Receive Capabilities Step 315(FIG. 3).

In some embodiments, Determine Needs Step 410 includes reading a cookie,IP address, or other identifying information on Client 110A. Thisidentifying information is then used to look up previously stored needsfor Client 110A. For example, if the needs of Client 110A werepreviously determined, these needs can be stored in a database in, forexample, Storage 155 or some location remote from Video Server System120. The identifying information can then be used to retrieve thesestored needs. Determine Needs Step 410 optionally includes bothretrieving stored information and performing some tests on Client 110A.For example, information regarding software available on Client 110A maybe stored while the connection between Client 110A and Video Source 130may be tested again.

In some embodiments, Determine Needs Step 410 is dependent on minimumrequirements set by a provider of a video game. For example, a gamepublisher may specify that a video stream be provided at least a minimumresolution and/or quality of service. This enables a game publisher toassure that a game demo meets minimum standards.

In a Determine Video Source Step 415 a Video Source 130 is selected forproviding streaming game video to Client 110A. The selected Video Source130 is typically selected from among a plurality of Video Sources 130.This plurality may include Video Sources 130 located at the samelocation or geographically distributed. The selection is based, in part,on the ability of the Video Source 130 to meet the needs of Client 110Aas determined in Determine Needs Step 410. For example, if a first VideoSource 130 can provide streaming video with a lower lag time relative toa second Video Source 130, the first Video Source 130 may be more likelyto be selected. Further details relating to the selection of VideoSources 130 are discussed elsewhere herein. For example, in relation toFIG. 7.

In a Select Game Logic Step 420 an instance of Game Logic 210 isselected for inclusion in the Video Source 130 selected in DetermineVideo Source Step 415. The Game Logic 210 is typically selected based onwhich video game the game player wishes to play. The Game Logic 210 mayalso be selected based on the needs of Client 110A. For example, ifClient 110A includes a touch screen interface, a version of the videogame (Game Logic 210) configured for use with a touch screen interfacemay be selected over a version configured for a keyboard interface. Insome embodiments, a demo or introductory version of a video game may beselected over a full version responsive to a user's account status,e.g., a user making a payment. An instance of Game Logic 210 isoptionally selected based on a need for a specific video frame aspectratio. Different instances of Game Logic 210 may be configured for thegeneration of video frames having different aspect ratios.

In a Provision Game Logic Step 425 the Game Logic 210 selected in SelectGame Logic Step 420 is provisioned onto the Video Source 130 selected inDetermine Video Source Step 415. Provision Game Logic Step 425 typicallyincludes copying computing instructions included within Game Logic 210from Storage 155 or from a code source external to Video Server System120 to working memory of Video Source 130. Provision Game Logic Step 425is optional where the selected Game Logic 210 is already provisioned onthe selected Video Source 130.

In a Provision (Virtual 3D Video) Driver Step 430 in instance of Virtual3D Video Driver 220 is provisioned to the instance of Video Source 130selected in Determine Video Source Step 415. The Virtual 3D Video Driver220 is typically retrieved from Storage 155 and copied to working memoryof Video Source 130. The Game Logic 210 provisioned in Provision GameLogic Step 425 and the Virtual 3D Video Driver 220 are also linkedtogether (e.g., a memory address of Virtual 3D Video Driver 220 isprovided to Game Logic 210) such that the game environment output ofGame Logic 210 is directed toward the Virtual 3D Video Driver 220.

In an optional Select Preprocessor Step 435 one or more Preprocessor 250are selected based on the needs of the Client 110 as determined inDetermine Needs Step 410. The selection may also be based on the type ofoutput of the Graphics Processing Unit 235 included within the selectedVideo Source 130. Select Preprocessor Step 435 is optional wherein rawgraphic provided by Graphics Processing Unit 235 is directly compatiblewith an encoder.

In a Select Encoder(s) Step 440, one or more video Encoders 255 areselected for inclusion in Video Source 130. One of this Encoders 255 istypically a preferred video encoder based on the video needs of theclient. Others of these encoders may be backup Encoders 255 in casethere is a need to change encoding during delivery of a video game. Forexample, a preferred Encoders 255 may produce a higher quality videostream than a backup Encoders 255 but the backup Encoders 255 mayrequire less bandwidth for delivery of video to Client 110A. Typically,the selection of Encoders 255 will be based on at least the videodecoders available on Client 110.

In an optional Select Post Processor(s) Step 445, one or more PostProcessors 260 are selected for inclusion in Video Source 130. Theselection is dependent on what further processing/packaging the outputof the Encoders 255 selected in Select Encoder(s) Step 440 require.

In a Provision DEP Step 450 a Dynamic Encoding Pipeline (Video DEP 245A)is provisioned in Video Source 130 using the Preprocessor 250, Encoders255 and/or Post Processors 260 selected in the previous steps. ProvisionDEP Step 450 includes linking these elements together such that they canpass access to video data received from Shared Memory 240 and/orGraphics Processing Unit 235. For example, in some embodiments memoryaddresses are provided to these elements so they can retrieve video datafrom Shared Memory 240 and then operating on that video data in a Heapwithin Video DEP 245A. This step may be performed under the control ofController 265.

Provision DEP Step 450 typically includes copying copies of theprovisioned elements from Storage 155 or from a code source discussedelsewhere herein. If more than one Encoder 255 is selected in SelectEncoder(s) Step 440, just the preferred Encoder 255 or alternativelymore than one Encoder 255 may be initially provisioned in Provision DEPStep 450. If only one encoder is initially provisioned additionalencoders may be provisioned at a later time. Video DEP 245A isoptionally configured in a tree structure. For example, one Preprocessor250 may provide video frames to one or more Encoders 255 and each ofEncoders 255 may provide video frames to one or more Post Processors260.

In some embodiments, the creation of Video DEP 245A in Provision DEPStep 450 includes receiving a command. This command includes parametersspecifying the elements to include in Video DEP 245A. Typically, a VideoDEP 245 is not allocated and provisioned until needed by a specificClient 110 at which time it is provisioned for that client.

FIG. 5 illustrates methods of serving multiple video games, according tovarious embodiments of the invention. In these methods Video Source 130supplies multiple video streams to multiple destinations. These videostreams are based on game environments generated by Game Logic 210 andare rendered using shared Graphics Processing Unit 235. The stepsillustrated in FIG. 5 may be performed in parallel to generate themultiple video streams. For example, Game Logic 210A, Game Logic 210Band Game Logic 210C may each be generating game environments forrendering and the resulting video streams can be sent to different gameClients 110. The operation of this Game Logic 210 mayor may not becoordinated. For example, in some embodiments the Game Logic 210 issynchronized so as to produce game environments at different times sothat Graphics Processing Unit 235 can render each game environment asquickly as possible.

In a Receive State Step 510, one or more game states are received byGame Logic 210. Each game state is directed to a different member ofGame Logic 210. For example, if Video Source 130 is configured toexecute a client (Game Logic 210) for World of Warcraft® using GameLogic 210A and a client for Grand Theft Auto® using Game Logic 210B,then game states for each of these games are received and directed tothe respective Game Logic 210. Each of the game states arerepresentative of a state of the respective video games.

The game states can be received from Game Server 125 or from a storagelocation within Video Source 130. In some embodiments, the receive gamestates are the result of game commands received by Game Logic 210A andGame Logic 210B and passed on to one or more Game Server 125, and/or theresult of game commands received by Game Server 125 from other gameplayers. In various embodiments a received game state represents a savedgame or a game state updated locally to Video Source 130.

In a Determine 3D Data Step 515, the game states received by instancesof Game Logic 210 are each used to determine a separate firstthree-dimensional game environment based on the game state and a playerpoint of view. The game environments are determined using rules of thegame as defined by each Game Logic 210. Each game environment is passedto a Virtual 3D Video Driver 220.

In a Render Step 520, the one or more three-dimensional gameenvironments generated in Determine 3D Data Step 515 are rendered tovideo data. This rendering is performed using shared Graphics ProcessingUnit 235. In some embodiments, the rendering of each three-dimensionalgame environments are done in parallel. For example, different gameenvironments may be processed by different parts of Graphics ProcessingUnit 235 at the same time, or Graphics Processing Unit 235 may beconfigured to process different game environments in different timeslices. These time slices mayor may not be long enough to generate asingle video frame.

In some embodiments, game environments are provided to GraphicsProcessing Unit 235 in a synchronized manner such that GraphicsProcessing Unit 235 can process one or a limited number of gameenvironments at the same time. The synchronization can be accomplishedby communication between members of Game Logic 210, or by control ofVirtual 3D Video Drivers 220. For example, Virtual 3D Video Drivers 220may be configured to report back completed processing of gameenvironments to each of Game Logic 210 in a synchronized manner suchthat the next game environment is provided from each of Game Logic 210at the best time. This synchronization is optionally managed byController 265. The result of Render Step 520 is typically a separateframe of raw video data for each receive game environment.

In a Transfer Step 525, the raw video data generated by GraphicsProcessing Unit 235 is transfer to Shared Memory 240. This transfer isoptionally performed under the control of Virtual 3D Video Drivers 220.The raw video data is optionally transferred to specific memorylocations within Shared Memory 240 assigned to specific Video DEP 245.Alternatively, the raw video data is transferred at specific timesassigned to specific Video DEPs 245. For example, each of Virtual 3DVideo Drivers 220 may be configured to control access by an assignedmember of Video DEPs 245 to Shared Memory 240 such that the member ofVideo DEPs 245 receives the correct video stream.

In an optional Preprocess Step 530, the raw video data generated byGraphics Processing Unit 235 and received by Video DEP 245A ispreprocessed using Preprocessor 250. Preprocess Step 530 typicallyincludes the conversion of the raw video data from RGB data to YUV data.Preprocess Step 530 is optional in embodiments wherein the raw videodata generated using Graphics Processing Unit is compatible withEncoder(s) 255. For example, if Graphics Processing Unit is configuredto generate YUV data directly. Multiple instances of Preprocess Step 530are optionally performed in parallel within different Video DEP 245 ondifferent video streams generated by Graphics Processing Unit 235.Preprocess Step 530 optionally further comprises changing a frame sizeof the raw video data.

In an Encode Step 535, one or more Encoders 255, e.g., the preferredEncoder 255, are used to encode the video data generated by GraphicsProcessing Unit 235 and optionally processed using Preprocessor 250. Theencoding is to one or more specific codec. In Encode Step 535 video datagenerated using Graphics Processing Unit 235 is optionally encoded tomore than one codec or to more than one frame size, color depth, orother video characteristic. For example, Encoder 255A may be configuredto encode according to one codec at a first frame size while Encoder2558 is configured to encode according to a second codec at a differentframe size.

In an optional Package Step 540, the encoded video frame generated inEncode Step 535 is packaged for communication to a Client 110, ifneeded.

In a Provide Step 545, the video frame encoded in Encode Step 535 andoptionally packaged in Package Step 540 is provided to one or moreClient 110 over Network 115 as part of a video stream. This video framemay also be provided to other destinations. For example, a copy of thevideo stream may be delivered to a website, a storage location, a socialnetworking site, an advertisement, and/or the like.

The steps illustrated in FIG. 5 are optionally performed in differentorders and may be repeated with variations. For example, Encoder 255Amay be used in one instance of Encode Step 535 while Encoder 2558 isused in another instance of Encode Step 535. This change may occurduring delivery of a video stream. Encoder 255A and Encoder 2558optionally communicate in order to maintain a sequence numbering of thevideo frames within the video stream. Reasons for changing encoding orother characteristics of the video stream are discussed elsewhereherein.

FIG. 6 illustrates various embodiments of Game System 100 including morethan one Video Server Systems 120. These Video Server Systems 120 areindividually identified as 120A, 1208, 120C, etc. Game System 100 caninclude at least 1, 2, 3, 5, 10, 25, 50, 100 or 500 instances of VideoServer System 120. These embodiments of Game System 100 further includea System Manager 615 configured to manage the use and operation of VideoServer Systems 120. These embodiments of Game System 100 optionally alsoinclude a Code Source 625 configured to provide game code to Clients 110at the same time these Clients 110 are receiving streaming video.

The members of the Video Server Systems 120 illustrated in FIG. 6 aretypically geographically dispersed. For example, Video Server System120A may be located in San Jose Calif., Video Server System 1208 may belocated in Tokyo, and Video Server system 120C can be located in NewYork City. The geographic distribution of Video Server Systems 120allows Clients 110 to receive video streams over a shorter geographicdistance that would occur if all of Video Server Systems 120 werelocated in the same place. To the extent that geographic distance isrelated to the time required to communicate between one of Clients 110and one of Video Server Systems 120, reducing the geographic distancereduces the latency of this communication.

Each of Video Server Systems 120 are optionally configured to providemultiple video streams to multiple Clients 110. These video streams canarise from different video games and/or multiple instances of the samevideo game. Different members of Video Server Systems 120 may beconfigured to provide streaming video from different sets of videogames. For example, Video Server System 1208 may be configured togenerate video streams arising from 20 instances of World of Warcraft®and 15 instance of Mortal Combat®, while Video Server System 1208 areconfigured to generate video streams arising from 20 instance of Worldof Warcraft and 25 instances of Spore®.

In various embodiments, Video Server Systems 120 are configured tocommunicate with Clients 110 and optionally one or more Game Servers 125via Network 115. Typically Network 115 includes the internet or otherpublic network that carries network traffic other than that related tothe output of Video Server Systems 120. Members of Video Server Systems120 optionally communicate through a Proxy Server 605 and a DedicatedConnection 610. Proxy Server 605 is a network server that is configuredto appear to be one of Video Server Systems 120 from the point of viewof Clients 110, and vice versa. The purpose of Proxy Server 605 is toforward received messages, e.g., from Client 110A to Video Server System120C and from Video Server System 120C to Client 110A. The advantage ofusing Proxy Server is that part of the communication can occur overDedicated Connection 610. Dedicated Connection 610 can have a highbandwidth dedicated to this communication. In one example, Proxy Server605 and Client 110A are both located in Amsterdam. Dedicated Connection610 is configured to enable high bandwidth communication between ProxyServer 605 and Video Server System 120C in New York City. Communicationbetween Video server System 120C and Client 110A occurs by firstcommunicating between Video Server System 120C and Proxy Server 605 viaDedicated Connection 610, and then by communicating between Client 110Aand Proxy Server 605 via Network 115. The use of Proxy Server 605 canallow a game player in New York to play on the same member of VideoServer Systems 120 as a game player in Amsterdam

System Manager 615 includes hardware, firmware, and/or software storedon a computer readable medium. For example, System Manager 615 caninclude an internet server having storage, an electronic processor, andsoftware configured to operate the electronic processor so as to performactions discussed herein. System Manager 615 may be located at a samelocation as one of Video Server Systems 120 or at a separate location.Typically, System Manager 615 includes at least an embodiment of ClientQualifier 160 and a Monitor 620.

The embodiment of Client Qualifier 160 within System Manager 615 isconfigured to qualify Clients 110 based on the capabilities of theClients 110 and the connections between Clients 110 and Video ServerSystems 120. When multiple Video Server Systems 120 are available toprovide video to Client 110A, Client Qualifier 160 optionally includescomputer instructions configured to first determine a geographiclocation of Client 110A based on mapping of the IP (internet protocol)address. This mapping is then used to identify one or more of VideoServer Systems 120 in geographic proximity to Client 110A. Following theidentification of nearby sources of streaming video, the communicationchannels between the identified members of Video Server Systems 120 aretested using further computing instructions of Client Qualifier 160.This testing is further described elsewhere herein. Client Qualifier 160is further configured to use the results of these tests to determinewhich of the Video Server Systems 120 is best qualified to deliverstreaming video to Client 110A.

Monitor 620 is configured to manage Video Server Systems 120. Forexample, Monitor 620 can track the usage and capacity of each of VideoServer Systems 120. This can include the geographic location of each ofVideo Server Systems 120, the number of video streams provided by each,the total amount of video data provided by each, the percentage of GPUand CPU cycles that are used, the amount of storage available,temperature, fan speed, hard drive and memory usage, network card usage,and/or the like. Monitor 620 can also track which games are provisionedon which of Video Server Systems 120 and how much each game is played.In some embodiments, Monitor 620 is configured to provision members ofVideo Server Systems 120 with Game Logic 210, Encoders 255, and/or othersoftware needed to provide video streams associated with specific games.Monitor 620 may also be configured to manage assignment of a specificmember of Video Server Systems 120 to provided streaming video to aspecific member of Clients 110, based on testing performed by ClientQualifier 160. This assignment can include reassignment from one memberto another member of Video Server Systems 120. Reassignment is desirableunder conditions discussed elsewhere herein.

Code Source 625 is configured to provide computing instructions to VideoServer Systems 120, Video Source 130, Clients 110, and/or other elementsof Game System 100. For example, Code Source 625 may be configured toprovide Game Logic 210, Preprocessor 250, Encoder 255, Post Processor260, Operating System 215, a software agent for testing the capabilitiesof Client 110, and/or the like. Code Source 625 may also be configuredto provide game code to Clients 110 as a background process as streamingvideo is being provided to these Clients 110. In typical embodiments,Code Source 625 includes a storage device such as a hard drive, a fileserver, a flash memory, an optical storage, or other device configuredto store computing instructions. Code Source 625 is optionallyconfigured to communicate with other parts of Game System 100 viaNetwork 115.

FIG. 7 illustrates methods of providing a video stream to a qualifiedclient, according to various embodiments of the invention. In thismethod one of Clients 110, e.g., Client 110A is qualified to receivevideo, based on the qualification a user of Client 110A is given anopportunity to request the video, and upon receiving the requeststreaming video is provided. The method is optionally performed usingGame System 100. For example, the steps illustrated in FIG. 7 can bemanaged by System Manager 615 and/or Controller 265. While the stepsillustrate in FIG. 7 are explained with reference to delivering a videostream to a single instance of Client 110. In alternative embodiments,the video stream may be delivered to multiple Clients 110. In theseembodiments, various steps such as tests may be performed with respectto each of the multiple Clients 110. The steps illustrated in FIG. 7 mayalso be adapted to the delivery of audio streams.

The methods illustrated in FIG. 7 start with Receive Request Step 300described elsewhere herein, e.g., with respect to FIG. 3. In this step arequest is received to display an object on Client 110A. This object isconfigured to give a game player an opportunity to request to receive astreaming video as part of playing a video game. The request could befor a specific video game, any of a class of video games, a “game of theday,” a game undergoing a sales promotion, the identity of a web page onwhich the object is displayed, or any video game. The request isoptionally received at System Manager 615 or one of Video Server Systems120.

In an optional Check Cookie Step 705, Client 110A is examined for abrowser cookie, an IP address, and/or other identifying information. Ifthis information is found it is used to retrieve previously determinedinformation regarding Client 110A. For example, if characteristics andneeds of Client 110A have already been determined and stored, theidentifying information is used to retrieve the stored characteristicsand needs. The information may be stored on System Manager 615 or one ofVideo Server System 120. The retrieval of previously stored informationmay eliminate the need to perform some or all tests in order to qualifyClient 110A to receive the object requested in Receive Request Step 300.

In an optional Provide (Web) Page Step 710 a web page is provided toClient 110A. This page mayor may not have the requested object includedtherein depending on the particular embodiment of the invention. If therequested object is included it is typically not fully functional.

In an Identify Servers Step 715 one or more Video Server Systems 120 areidentified as potential candidates for providing streaming video toClient 110A. The identification of these servers may be based on theirproximity to Client 110A. For example, an IP address of Client 110A maybe mapped in order to determine its approximate location. Theidentification of these servers may also be based on an initial pingtest between the various Video Server Systems 120 and Client 110A. Theidentification of these servers may also be based on the availablecapacity of Video Server Systems 120. For example, an instance of VideoServer Systems 120 may not currently have capacity to deliver thestreaming video to Client 110A. In this case this instance of VideoServer Systems 120 is not considered as a candidate. It may occur thatno Video Server Systems 120 are identified as candidates for providingstreaming video to Client 110A. In this case, Client 110A cannot bequalified to receive the streaming video and the method is halted.

In a Test Client Step 720, Client 110A and the connections betweenClient 110A and the various Video Server Systems 120 identified ascandidates in Identify Servers Step 715 are tested. Test Client Step 720can include Provide Agent Step 305, Execute Agent Step 310 and ReceiveCapabilities Step 315, as described elsewhere herein. The tests made inTest Client Step 720 are optionally performed using Java® code. Thiscode can be included in a browser plug in or embedded in a web page. Forexample, in some embodiments, the testing is performed just using Javacode embedded in a web page.

The testing performed in Test Client Step 720 can include: latency testssuch as the time required to communicate between Client 110A and each ofthe candidate Video Server Systems 120; bandwidth tests based on, forexample, HTTP or UDP; tests of the CPU on Client 110A, tests configuredto determine the availability of plug-ins on Client 110A (plug-ins caninclude testing agents, decoders, or the like); tests configured todetermine the operating system running on Client 110A; tests configuredto determine the types of devices included in or attached to Client110A, such as display devices or graphics processing units; testsconfigured to determine the size and aspect ratio of an area on Client110A on which streaming video is to be displayed; tests configured todetermine a maximum packet size that can be transmitted to Client 110Awithout being split (e.g., a maximum transmission unit); testsconfigured to determine if a Wifi connection is included in thecommunication channels between Client 110A and each of the candidateVideo Server Systems 120; and tests configured to determine thereliability (e.g., jitter, packet loss and variation in bandwidth)between Client 110A and each of the candidate Video Server Systems 120.

The tests performed in Test Client Step 720 are optionally performedover multiple web page loads. These can occur over time, e.g., overdays. In order to perform tests over time identification information,such as that identified in Check Cookie Step 705 is used to associateprevious test data with new test data. For example, some reliabilitytesting of communication channels between Client 110A and candidateVideo Server Systems 120 may occur each time a game player interactswith Game System 100 and the results of these tests can then be combinedto create a statistically reliable estimate of the reliability. Teststhat do not take significant time, e.g., ping tests, are typicallyperformed on a single page load. While tests that take more time, e.g.,bandwidth and reliability tests, are optionally performed over severalpage loads or several instances of Test Client Step 720. Old test datais optionally allowed to expire such that testing reflects currentconditions of Client 110A. Test Client Step 720 is optionally performed,in part, after delivery of streaming video to Client 110A has begun.

In a Determine Server Availability Step 725 the availability of thecandidate Video Server Systems 120 identified in Identify Servers Step715 to provide the streaming video are determined. This available may bedependent on resources available on each Video Serving System 120. Forexample, whether sufficient Processor 150 or Graphics Processing Unit235 capacity is available. If the request received in Receive RequestStep 300 is directed toward a specific video game, then Determine ServerAvailability Step 725 can include confirming that the candidate VideoServing Systems 120 are configured to provide streaming video from thatspecific video game. An instance of Video Serving Systems 120 may beprovisioned with required Game Logic 210 if needed.

In a Determine Game Availability Step 730 the availability of a specificvideo game is determined. If the request received in Receive RequestStep 300 is directed toward a specific video game then the availabilityof that game is determined. Otherwise, the availability of several gamesmay be determined until an available game is found. Game availabilitymay also be dependent on licensing of game code from a publisher or agame promotion program.

The availability of a specific video game may be dependent on testresults generated in Test Client Step 720. For example, some video gamesrequire a greater amount of bandwidth and or shorter ping time thanother video games. Optionally, if the tests results show that a computergame cannot be delivered with a minimum level of quality then that gameis not available to Client 110A, at this time. A game player usingClient 110A, a manager of Game System 100, and/or a publisher of a videogame may determine the minimum level of quality. The minimum level ofquality can include several parameters such as lag time, bandwidth,reliability, and frame size.

In a Select Server Step 735 one of the candidate Video Serving Systems120 is selected to provide the video game, as streaming video, to Client110A. This selection is based on the results of Test Client Step 720,Determine Server Availability Step 725, and/or Determine GameAvailability Step 730. Typically the Video Serving System 120 best ableto provide the streaming video to Client 110A is selected. Select ServerStep 735 optionally includes temporarily reserving capacity (memory, CPUand GPU, etc.) on the selected Video Server System 120. This assuresthat the required resources are available if the video stream isactually delivered. Select Server Step 735 is an alternative embodimentof Determine Video Server Step 415 (FIG. 4).

In a Determine Needs Step 740 the needs of Client 110A are determined.Determine Needs Step 740 often includes merely retrieving informationfound in Test Client Step 720 and/or Determine Game Availability Step730. For example, the need of a specific decoder can be determined basedon the results of Test Client Step 720 and a minimum frame size can bedetermined based on the results of Determine Game Availability Step 730.Determine Needs Step 740 is an alternative embodiment of Determine NeedsStep 410.

In a Provision Step 745 the selected Video Serving System 120 (and aVideo Source 130 therein) are provisioned to provide the streaming videoto Client 110A. Provision Step 745 optionally includes Select Game LogicStep 420, Provision Game Logic Step 425, Provision Driver Step 430,Select Preprocessor Step 435, Select Encoder Step 440, Select PostProcessor Step 445, and/or Provision DEP Step 450, as discussedelsewhere herein. The provisioned logic and other computing instructionsare optionally provisioned from Code Source 625.

In an Offer Game Step 750 the object requested in Receive Request Step300 is provided to a game player on Client 110A. Offer Game Step 750 caninclude providing a pop-up browser window, changing an area of a webpage already provided in Provide Page Step 710, sending an e-mail ortext (SMS or MMS) message, sending a link, or direct streaming of gamevideo. For example, in some embodiments a region of the web pageprovided in Provide Page Step 710 directly receives streaming game videowith no further actions by the game player. The region within the webpage can show, for example, streaming video of the starting line ofMario Kart, or the character selection screen or a starting zone ofWorld of Warcraft. The game player can begin game play by just selectingthe region and starting to provide game commands. Optionally, the regionof the web page can show streaming video including the avatar of afriend of the game player playing the game and an avatar of the gameplayer. This allows the game player to instantly join a game with afriend. Friends can be identified using relationships established onsocial networking sites, for example. In some embodiments, Offer GameStep 750 is delayed until capacity to provide the streaming videobecomes available. Offer Game Step 750 is optionally automaticallyperformed once the capacity is available.

Offer Game Step 750 is contingent on Client 110A being qualified toreceive streaming video of the game and that the game be available.Offer Game Step 750 is typically automatic once these conditions havebeen satisfied. In some embodiments, Offer Game Step 750 includes addingfunctionality to a web page already displayed to a game player inProvide Page Step 710. In these embodiments the requested object isalready displayed but not fully functional. The added functionalitymakes it functional. The added functionality may be a proper link to asource of streaming video.

In a Deliver Game Step 755 the video stream is provided from theselected Video Server System 120 to Client 110A. Deliver Game Step 755includes changing the contents of the video stream responsive to gamecommands received from the game player via Client 110A. Deliver GameStep 755 optionally includes providing one or more copies of the videostream to other destinations, e.g., to other IP addresses. These copiescan have different characteristics than the video stream provided toClient 110A, but are based on the same game environment provided toGraphics Processing Unit 235. The other destinations include websites,storage locations, social networking sites, other Clients 110, and/orthe like. The streaming video is optionally provided to all game players(and lurkers) within the same game room. A game room is a virtual roomin which players can join, leave and play games as a group. A game roomcan include game players and optionally lurkers who observe but to notplay the video game. Game rooms are useful when playing games designedfor a group of players sitting around a game console and sharing adisplay device.

Deliver Game Step 755 optionally includes delivery of information otherthan the video stream to Client 110A. For example, Deliver Game Step 755can include deliver of executable game code to Client 110A as abackground process. This executable game code can allow eventualexecution of the video game in a client side mode (where renderingoccurs on Client 110A) rather than at Video Source 130.

In an optional Monitor Step 760 delivery of the streaming video toClient 110A is monitored. This monitoring can include making sure thatthe delivery of the streaming video meets a required quality of service.The monitoring can also include gathering further data characterizingthe reliability and bandwidth of the communication channel betweenClient 110A and the selected Video Server System 120. In someembodiments, Monitor Step 760 includes monitoring for changes in thesize of a browser window in which the streaming video is being presentedon Client 110A. For example, if the game player changes the size of abrowser window the monitoring may determine that a new frame size forthe streaming video is warranted.

In an optional Add Game Player Step 765 a new game player anddestination (e.g., Client 110) for the streaming video is added. Thismay occur, for example, when a new player enters a game room. Theaddition of a new game player may change the geographical distributionto which the streaming video is delivered. When this occurs, IdentifyServers Step 715, Test Client Step 720, Determine Server AvailabilityStep 725, Determine Game Availability Step 730 and/or Select Server Step735 may be repeated.

In an optional Change Server Step 770 delivery of the streaming videofrom the member of Video Server Systems 120 selected in Select ServerStep 735 is transferred to a different member of Video Server Systems120. This transfer may include provisioning the appropriate resources onthe new Video Server System 120, optionally synchronizing execution ofGame Logic 210 executing on the two Video Server Systems 120, andtransferring delivery of the video stream. Further details of thisprocess are disclose in U.S. patent application Ser. No. 12/790,955filed May 31, 2010 and entitled “Game Execution Environments.”

Using the methods illustrated in FIG. 7 the utilization of Video ServerSystems 120 can be controlled. For example, as game players are notoffered to play video games unless there is capacity available, sparecapacity is not necessary. This allows for a high utilization rate(e.g., greater than 50, 75, 80, 90 or 95%) of Video Server Systems 120.In various embodiments, more than 80, 60 or 50% of requests received inReceive Request Step 300 are denied, even with more than ten of VideoServer System 120. While the examples provided herein are primarilydirected at the delivery of video, the systems and methods describedherein can be applied to the generation of still images and the deliveryof still images, and/or the delivery of audio frames and the delivery ofaudio streams. In some embodiments, both video and audio streams aregenerated and provided as “audio-visual” streams.

In some embodiments, one or more Video Server Systems 120 are operatedin multiple modes. In a “game play” mode resources of Video ServerSystems 120 are used to provide video games to video game players attimes selected by the video game players, i.e., video games are providedon the game players demand. In this mode players can request gameswithout regard to resource availability. At the same time, in a “demo”mode video game demos are provided at times controlled by the VideoServer Systems 120, i.e., game demos (that typically include playing thevideo game) are provided only when offered to a game player. Offers areonly made when resources are available for play and, thus, a player canonly request a game when resources are available. Resources of the VideoServer Systems 120 are allocated to the demo mode so as to minimizespare resource capacity and thus maximize utilization. The number ofdemos offered is dynamically adjusted according to the resources thatcan be allocated to the demo mode. Using this approach one or several ofVideo Server System 120 can include the resources needed to support themaximum number of simultaneous users expected in the game play mode, andat times when less than the maximum number of simultaneous user arebeing supported, demos are offered to fill the access capacity. Thedemos can be of different or the same video games. Systems and methodsof serving demos are further described in U.S. patent application“Dual-Mode Program Execution” and having Ser. No. 13/231,790. Balancingof the game play mode and demo mode is optionally coordinated usingelements of System Manager 615 or “Game Delivery Systems 315” asdescribed in the U.S. patent application “Game Execution Environments”and having Ser. No. 12/790,955. The above U.S. patent applications arehereby incorporated herein by reference.

FIG. 8 illustrates overlays on game video (a video frame), according tovarious embodiments of the invention. Three Overlays 810A, 810B and 810C(collectively identified as Overlays 810) have been applied to the videogenerated by the video game. These Overlays 810 may be applied as asingle overlay or as several separate overlays. Each of Overlays 810represent input controls and are clickable objects associated with anunderlying input command. For example, the “TAB” Overlay 810C may bepart of an input control that generates a set of screen coordinates whenclicked, the screen coordinates being mapped (at Client 110A or VideoServer System 120) to a “TAB” keystroke. In this example, the screencoordinates and the “TAB” keystroke are both examples of input commands.FIG. 8 further includes examples of Input Controls 815 that are notoverlays, but are included in the video frame by Game Logic 210A. Theseclickable Input Controls 815 are configured to generate different gamecommands. In some embodiments, an overlay includes an image of a cursor.

Each of the Overlays 810 represents a different game input control asindicated by the text therein. Overlay 810A represents a carriagereturn, Overlay 810B represents an escape key button, and Overlay 810Crepresents a Tab key button. In typical embodiments, these Overlays 810are associated with specific regions of the video frame and thesespecific regions are mapped to the represented game input controls. Forexample, the region of the image occupied by Overlay 810C is mapped tothe “Tab” key such that clicking on or otherwise making a selection inthis region is interpreted as a Tab keystroke. Selection can beaccomplished by using a pointing device such a joystick or mouse, or bytouching this region on a touch sensitive screen.

FIG. 8 further includes an Overlay 820 representing a game menu, such asmay be used to manage a game room or other aspect of game execution.Overlay 820 includes a plurality of active regions mapped to differentgame commands (inputs). For example, the area occupied by the word“Search” within Overlay 820 is mapped to a search command while the areaoccupied by the word “Join” is mapped to a command to join a game room.

FIG. 8 further includes an Overlay 830 including an image of a gameplayer. Overlay 830 may be a static or real-time image of another gameplayer. For example, if the video frame illustrated in FIG. 8 is to besent to Client 110A then Overlay 830 may include a real-time video frameof a player of Client 110B. Overlay 830 may alternatively include a viewof the video game from another player's point of view, a view from asecond point of view for a same player, or a view of some real-lifeevent. In one example, Overlay 830 includes a video frame based on asecond point of view of the same player whose first point of view wasused to generate the underlying image. This second point of view can begenerated using a second Video DEP 245. The number, shape, size andcharacteristics of overlays that may be placed over video data can varywidely. Overlays mayor may not be mapped to game commands. Overlays canbe applied either server and/or client side. Some embodiments includeoverlays applied both server and client side.

The mapping of user input controls to game commands is not limited toapplications in which Clients 110 include a touch screen or pointingdevice. FIGS. 9A, 9B and 9C illustrate game control mappings betweenButtons 910 of a Game Controller 920 and game commands, according tovarious embodiments of the invention. These game commands include attackactions, movement actions, changing views, menu display request items,cursor control commands, targeting commands, save actions, video settingmenu items, print screen, and/or the like.

The mapping of user input controls to specific game input commands canoccur at Clients 110 or at Video Server System 120. For example, theiPad® from Apple Computer, Inc. is configured to convert touches to itsscreen, movements, and combinations of touches to specific commands andfurther have the capacity to remap these commands. The iPad and iPhone®are also configured to convert movements of the device itself tospecific commands. However, other types of Client 110 may not have thesecapabilities, in which case the mapping can occur at Video Server System120.

Mapping of user input controls to game commands (and, more generallyinput commands to other input commands) is optionally performed using alook-up table or other type of input mapping. FIG. 10 illustrates partof an Input Map 1010 configured for mapping of user input controls togame commands, according to various embodiments of the invention.Typically, an Input Map 1010 for command mapping will include moreentries than illustrated. The mappings within Input Map 1010 areoptionally dependent on a state of the video game, as described furtherelsewhere herein. Further, mappings within Input Map 1010 are optionallydependent on characteristics of a game player's account, e.g., a premiumaccount may be allowed commands that a standard account is not.

The first mapping illustrated in Input Map 1010 is a direct character tocharacter map of “D” to UK.” This type of mapping may be useful, forexample, in games that are configured for several players to share akeyboard, each using their own subset of the keys. When such a game isplayed by geographically disperse game players each player may use thesame keys but one of the player's inputs are mapped to a different partof the keyboard.

The second mapping illustrated in Input Map 1010 includes mapping of ascreen area to a key. Specifically, a rectangle with corners at(100,200) and (400,500) is mapped to the F1 function key. Thecoordinates can be in image, window or screen pixels. This type ofmapping is useful when a touch screen or pointing device is used as aninput. This type of mapping can be used to map images within an overlayto specific game commands. This type of mapping can also be dependent onmovement of a finger or cursor on a screen, or the number of points atwhich a touch screen is touched.

The third type of mapping illustrated in Input Map 1010 includes mappingof a controller button to a key. Specifically, a “Red button” is mappedto a “Spacebar.” This type of mapping can be used to map game controllerinputs, such as those illustrated in FIGS. 9A-9C, to specific gamecommands.

The fourth type of mapping illustrated in Input Map 1010 includesmapping motion (“Right tilt”) of an input device to a key. This type ofmapping can be used with joysticks, iPhones, iPads, the Nintendo Wii®,or other types of Clients 110 with motion or location sensingcapabilities.

The fifth type of mapping illustrated in Input Map 1010 includes mappingof a movement of an object as detected by an external device, such as acamera, to a key “A.” This type of mapping is optionally used in Clients110 that include radio or video motion detection systems. Using thistype of mapping allows games designed to be played using a keyboard tobe played using new types of input devices, such as the Kinect® fromMicrosoft.

A sixth type of mapping may be dynamic. For example, in embodimentswherein motion of a finger on a touch screen is mapped to joystickmovements, where the finger first touches the screen may be dynamicallymapped to the joystick “center” position. Screen positions just above,below, right, left, etc. of the first touch position are mapped tojoystick positions up, down, right, left, etc., respectively. As such, ajoystick movement can be entered by first placing a finger on the screenand then sliding the finger in some direction. Dynamic mapping may alsoinclude multi-finger touches to the screen. For example, a touch on afirst point of the screen can determine the location and orientation ofa mapping used to convert a second touch to a game command. Dynamicmapping is optionally facilitated by an overlay illustrating themapping. For example, a semi-transparent joystick may be placed at theposition the finger is first placed in the above example. In this casethe second touch optionally represents movement of that joystick.

FIG. 11 illustrates methods of using an overlay, according to variousembodiments of the invention. The overlay can be audio and or video. Inthe method illustrated, a video frame is generated using GraphicsProcessing Unit 235 and an overlay is added to this video frame prior toencoding. The methods illustrated by FIG. 11 are optionally performed inparallel using several of Game Logic 210, several of Video DEP 245and/or several of Audio DEP 270.

Specifically, in a Generate Step 1110 a game environment is generatedbased on game rules and game commands received via Network 115 andClients 110 from one or more game players. For example, a game player atClient 110A and a game player at Client 110A may each provide a gamecommand that is used to update a game state from which the gameenvironment is generated. Generate Step 1110 may include Receive StateStep 510 and Determine 3D Data Step 515, as discussed elsewhere herein.In some embodiments, Generate Step 110 includes generating audio databased on events within the game, the game rules and/or received gamecommands.

In an optional Render Step 1120 the game environment generated inGenerate Step 1110 is rendered using Graphics Processing Unit 235.Render Step 1120 is an embodiment of Render Step 520 discussed elsewhereherein. The result of Render Step 1120 is a video frame. Render Step1120 is optional in embodiments where the result of Generate Step 110included only audio data and/or did not include a 3D game environment.

In a Provide Step 1130 the video frame generated in Render step 1120and/or the audio data generated in Generate Step 1110 is provided to oneor more encoding pipelines such as Video DEP 245A and/or Audio DEP 270A.For example, a video frame may be provided to Video DEP 245A while anaudio frame is provided to Audio DEP 270A. Provide Step 1130 is anembodiment of Transfer Step 525.

In an Add Overlay Step 1140 a video overlay is added to the video frameprovided in Provide Step 1130, and/or and audio overlay is added toaudio data provided in Provide Step 1130. Add Overlay Step 1140 istypically performed using Overlay Logic 290 and/or Overlay Logic 295, asdescribed elsewhere herein. More than one video overlay and more thanone audio overlay are optionally added in Add Overlay Step 1140. Theaudio overlay optionally includes audio data received from one or moreof Clients 110 as packets via Network 115. For example, the audiooverlay can include voice data of game players received from two, threeor more of Clients 110. The identities of the added overlay(s) areoptionally dependent on a state of the video game.

In some embodiments Video Source 130 receives audio data from several,e.g., more than 3, 5 or 10 Clients 110 and Add Overlay Step 1140includes determining which of the audio data should be used in aparticular audio overlay. For example, while 10 game players may beusing Video Source 130 only the voice data of two or three of theseplayers may be included in the audio overlay. The determination of whichaudio data to include in an overlay is optionally based on a socialrelationship between game players. For example, one game player mayselect which whose voice data he wishes to receive. The socialrelationships can include membership in a guild, party, being in a samebattleground, or game instance, having avatars located near each otherin the same game environment. In some embodiments, the volume of audiodata is adjusted responsive to an in game distance between two avatars.

In an Encode Step 1150 the video frame and/or audio data, as well as anyoverlays are encoded. This is typically accomplished using Encoder 255and/or Audio Encoder 280. Encode Step 1150 is an embodiment of EncodeStep 535. Encode Step 1150 may be performed before or after Add OverlayStep 1140.

In a Deliver Step 1160 the video frame and/or audio data encoded inEncode Step 1150 is provided to one of Clients 110 via Network 115. Thevideo frame and/or audio data are optionally provided to more than onelocation. For example, the same data can be provided to more than onegame player. Deliver Step 1160 is an embodiment of Provide Step 545.

In an optional Map Step 1170 part of a video overlay applied in AddOverlay Step 1140 is mapped or remapped to a game command. For example,a region of the video frame to which the video overlay was added can beset in an Input Map 1010 as mapping to a specific game command. Anoverlay can be applied to more than one region of a display or a videoframe. Each region is optionally mapped to a different game command.Input Map 1010 can be located on Video Server System 120 or one ofClients 110.

In an optional Receive Step 1180 a mapped game command is received at orwithin Video Source 130. The mapped game command is optionally a commandthat has been mapped from a region of a video frame to which an overlaywas applied. The received game command may have been mapped at eitherone of Clients 110 or at Video Server System 120. The received gamecommand is optionally used to update a state of the video game.

FIG. 12 illustrates parts of Video Server System 120 including SentinelLogic 1210, according to various embodiments of the invention. In theseembodiments undesirable input commands and input controls are blocked.Undesirable input commands and input controls include, for example,those whose operation expects that Game Logic 210A is executing onClient 110A rather than on Video Server System 120. For example, aninput command to select local video settings, manage local add-ons, savea game locally, manage audio settings, refresh a browser window, open aUniversal Resource Locator in a browser window, etc. are potentiallyundesirable. These input commands can be redirected or otherwise blockedusing the embodiments of Video Server System 120 illustrated in FIG. 12.Input commands that should be redirected or otherwise blocked arereferred to herein as “blocked input commands.” In some embodiments, theidentities of blocked input commands are dependent on the state of avideo game and/or on a game player's account.

Some blocked input commands are associated with input controls includedin the encoded video provided by Video Source 130 to Client 110A. Whenan input command is blocked it is sometimes desirable to modify theassociated input control such that a game player is not presented withan inoperable or improper control. In other words, if add-ons are notallowed then the game player should not be shown a “load add-ons”button. Input controls associated with blocked input commands can begrayed out, covered completely, changed to represent a different inputcommand, or otherwise modified. Modified input controls are referred toherein as “blocked input controls.” Regarding both commands and inputcontrols, the term “blocked” is meant to include generally both itemsthat are completely eliminated as well as those that are merelyredirected or changed. Blocked input controls are optionally animated ormoving and can be moving controls in 20, 3D or stereoscopic 3D.

Sentinel Logic 1210 is configured to detect the presence of blockedinput controls. Detection is typically accomplished by monitoring theoutput of one or more elements of Video Server System 120 and comparingthis output with a set of objects known to represent blocked inputcontrols. This detection can occur at several points in the generationof encoded video (or still images). The set of objects is optionallystored in and retrieved from Storage 155. In some embodiments themonitoring for blocked input controls occurs only ever nth frame orimage (n>1). In some embodiments monitoring occurs only when the stateof a game changes and/or a new image is generated. Sentinel Logic 1210includes hardware, firmware and/or software stored on a computerreadable medium.

In one example, detection occurs in a game environment generated by GameLogic 210. This detection is accomplished by monitoring the generatedgame environments for text, shapes or textures known to representblocked input controls. In another example, detection occurs at theoutput of Graphics Processing Unit 235, e.g., in memory of GPU 235 or inShared Memory 240. At this point, detection is accomplished by comparingthe output with a bitmap known to represent blocked input controls. Thebitmap may be compared with specific regions within the output or withall of the output. Thus, a blocked input command that could appear in avariety of alternative positions can be detected. Detection may,likewise, occur at any point within Video DEP 245A. For example,Sentinel Logic 1210 can be configured to detect the presence of ablocked input control at the output of Overlay Logic 290. Sentinel Logic1210 is optionally further configured to detect the presence of ablocked input control in an overlay before the overlay is applied to anoutput of Graphics Processing Unit 235. Sentinel Logic 1210 isoptionally further configured to detect the presence of a blocked inputcontrol in video frames or images received from sources other thanGraphics Processing Unit 235.

The operation of Sentinel Logic 1210 may be responsive to optional StateLogic 1220. State Logic 1220 is configured to monitor the state of avideo game (or some other computer application) and identify which inputcontrols are blocked input controls based on the state. A specific inputcontrol may be a blocked input control in one state and a non-blockedinput control in another state of the computer game. For example, a“save button” may be blocked when displayed on a top level menu andconfigured to initiate a command that would save the game locally, butunblocked when displayed on an avatar configuration menu and configuredto initiate a command that would save an avatar configuration. Differentsets of input controls can, thus, be blocked or non-blocked depending onthe state of the video game. Sentinel Logic 1210 is configured to usethe current set of blocked input controls when detecting the presence ofblocked input controls. Further, the blocking of input controls and/orwhich input commands are blocked is optionally dependent oncharacteristics of a game player's account, e.g., a premium account maybe allowed controls and commands that a standard account is not.

The embodiments of Video Server System 120 illustrated in FIG. 12optionally further include Command Processing Logic 1230. CommandProcessing Logic 1230 is configured to redirect or otherwise blockcommunication of blocked input commands to members of Game Logic 210. Insome instances, Command Processing Logic 1230 is also configured toredirect or otherwise block communication of blocked input commands toadd-ons. As with Sentinel Logic 1210, operation of Command ProcessingLogic 1230 is optionally controlled by State Logic 1220. Thus, the setof input commands that are blocked by Command Processing Logic 1230 canbe dependent on a video game state. Command Processing Logic 1230includes hardware, firmware and/or software stored on a computerreadable medium. Command Processing Logic 1230 is optionally disposed toexamine input commands received by I/O Device 140 before they arecommunicated to parts of Video Source 130.

Blocked input commands can be initiated by an input control included inthe video or image output of Video Server System 120, by keystrokes, bya browser add-on executing on Client 110A, by game logic executing onClient 110A, by a video game add-on, by a game controller, by a motiondetection system, by a voice recognition system, or by any other inputdevice. Blocked input commands can be blocked completely by CommandProcessing Logic 1230. Alternatively, Command Processing Logic 1230 maybe configured to map a blocked input command to a non-blocked inputcommand or to redirect a blocked input command to a new destination.Input commands that are completely blocked are dropped without furtherprocessing. Redirected input commands can be redirected to an add-on orto alternative game logic. For example, a “save” command directed at apart of Game Logic 210A configured to save a game on a client may beredirected to alternative game logic configured to save the game on aVideo Server System 120. Input commands that are remapped are optionallyremapped using a mapping similar to that illustrated in FIGS. 9A, 9B and9C. Under the control of State Logic 1220, blocked input commands can becompletely blocked in a first game state, redirected to a first locationin a second game state, redirected to a second location in a second gamestate, remapped to a first alternative input command in a third state,remapped to a second alternative input command in a forth state, or anycombination of the above.

In some embodiments, a blocked input control is detected by SentinelLogic 1210 in the output of GPU 235 within Shared Memory 240. Theblocked input control includes a “Load Add-on” button and a data entryfield configured for a game player to enter a local file name and pathof an add-on. Overlay Logic 290 is used to modify the “Load Add-on”button to read “Load Remote Add-on” and the date entry field is coveredusing a background colored bitmap. When the “Load Remote Add-on” buttonis clicked by a game player a blocked “Load Add-on” input command isgenerated at Client 110A and sent to Video Server System 120. This inputcommand can be communicated as a text stream, as a screen coordinate, asa binary object, and/or the like. At Video Server System 120 the “LoadAdd-on” input command is redirected by Command Processing Logic 1230 togame logic configured to load one of a set of pre-approved add-ons onVideo Server System 120. This game logic is optionally itself an add-onto Game Logic 210A. In some embodiments, an audio overlay including forexample a “warning” sound is added to the audio stream at the same timethat a video overlay is added to a video frame or when a blocked inputcontrol is activated by a game player.

In some embodiments, some blocked input commands are configured tocontrol the operation system or some other software application ofClient 110A. In these embodiments the blocked input command isredirected to optional Client Control Logic 1240. Client Control Logic1240 includes hardware, firmware, and/or software stored on a computerreadable medium, and can be disposed on Video Server System 120 and/orone of Clients 110. In some embodiments Client Control Logic 1240includes a browser add-on configured to be added to a browser of Clients110.

In some embodiments, Client Control Logic 1240 is configured to openand/or close windows, e.g., browser windows (or tabs) or other windowtypes, on Client 110A. These windows are optionally provided with anetwork address or universal resource locator configured to retrieveimages or video from an instance of Video Server System 120. As such,Client Control Logic 1240 can be used to open a new window on Client110A and automatically provide a website and/or streaming video to thatwindow. The opened window is optionally configured for logging into asocial networking web site or configured for local execution of aversion of Game Logic 210A.

In one example, Command Processing Logic 1230 is configured to redirecta blocked input command to open a Windows Explorer® window on Client110A such that a user can select a local file. This blocked inputcommand is redirected to Client Control Logic 1240 (optionally amongother places) which opens a browser window on Client 110A rather than aWindows Explorer window. The browser window is provided with a URL to avideo output of Video Server System 120, e.g. a port of I/O Device 145,at which a video of a virtual file system of Video Server System 120 isprovided. The user (or game player) can navigate the virtual file systemto select a file by clicking on input controls within the providedvideo. Either Client Control Logic 1240 or Command Processing Logic 1230may be configured to initiate generation of the video of the virtualfile system using video Server system 120. A result of this process isthat a second browser window is opened and the images (e.g., video)provided to that window gives the same look and feel as if the videogame or other software application were executed locally on Client 110A.

Video Server System 120 optionally further includes Discovery Logic 1250configured to identify blocked input controls within graphics output ofVideo Source 130. Once identified these blocked input controls can bemodified in future output of Video Source 130 using the systems andmethods discussed herein. The operation of Discovery Logic 1250 isoptionally automatic. Discovery Logic 1250 includes hardware, firmware,and/or software stored on a computer readable medium.

Discovery Logic 1250 may be configured to identify blocked inputcontrols using a variety of different approaches. In some embodiments,Discovery Logic 1250 compares input commands received from Client 110Awith a list of undesirable input commands. When a match is found theblocked input control on video already provided to Client 110A isidentified, for example, by matching a bitmap, optical characterrecognition, or use of the coordinates of a game player's click on thevideo. In some cases the full identification of a blocked input controlrequires the identification of multiple click locations.

In some embodiments, Discovery Logic 1250 monitors activities of GameLogic 210A in order to determine which input commands are blocked inputcommands. For example, Discovery Logic 1250 may watch for calls made byGame Logic 210A to a file system or to video hardware, and observe whichinput commands were recently received by Game Logic 210A that may havecaused these calls. Once the blocked input commands are identified, theassociated blocked input control can be identified as describedelsewhere herein. In some embodiments, Discovery Logic 1250 isconfigured to examine the output of add-ons.

FIG. 13 illustrates methods of generating game video, according tovarious embodiments of the invention. These methods are optionallyperformed using elements of the embodiments illustrated in FIG. 12. Thegenerated game video includes an overlay configured to modify a blockedinput control.

In Provision DEP Step 450, an encoding pipeline such as Video DEP 245Ais provisioned. Provision DEP Step 450 is optional in embodimentswherein an encoding pipeline, including overlay logic, is alreadyavailable. Provision DEP Step 450 may include other steps illustrated inFIG. 4. Further details of Provision DEP Step 450 are describedelsewhere herein.

As described elsewhere herein, Generate Step 1110 includes generating agame environment based on game rules and input commands received fromone or more of Clients 110. The game environment is rendered to videoframes using Graphics Processing Unit 235 in Render Step 1120 and thesevideo frames are provided to one or more of Clients 110 in Provide Step1130.

In a Monitor Step 1315 the video frames are monitored for the presenceof blocked input controls. This monitoring is optionally performed usingSentinel Logic 1210. The monitoring can occur prior to the inputs toGraphics Processing Unit 235, e.g., by monitoring the game environments.Alternatively, or additionally, the monitoring can occur by examiningvideo frames generated by Graphics Processing Unit 235. The frames canbe monitored at memory internal to Graphics Processing Unit 235, at theoutputs of Graphics Processing Unit 235, within Shared Memory 240,within Video DEP 245A, and/or the like. For example, monitoring mayoccur at Shared Memory 240 and also after the inputs to Overlay Logic290. In some embodiments only every nth (n>1) frame is monitored. Framesare optionally monitored by searching for image characteristics,features and/or bitmaps known to represent blocked input controls and,thus, indicate a need to block input controls, certain keystrokes orother inputs when the frame is displayed. As described elsewhere herein,State Logic 1220 is optionally used to control the monitoring based on agame state. In various embodiments, using the game state allows thenumber of bitmaps that are monitored for at a time to be reduced by atleast 50, 75, or 90%. Monitor Step 1315 is optionally applied to videoframes or still images that are generated or retrieved through meansother than Graphics Processing Unit 235. For example, Monitor Step 1315can be applied to two dimensional images, video retrieved from storage,live video generated using a camera, video received from a member ofClients 110, video received from an external source, and/or the like.

In optional Add Overlay Step 1140 an overlay is applied to an image(e.g., video frame) using Overlay Logic 290. As described elsewhereherein, the overlay can include input controls. The overlay isoptionally therefore also monitored (in Monitor Step 1315) for thepresence of blocked input controls. The overlay can be monitored beforeor after it is applied to the image.

In Encode Video Step 1150 and Deliver Step 1160 video frames are encodedusing one or more of Encoders 255 and I/O Device 145. The video framesare eventually received for display on one or more of Clients 110. Thesesteps are described further elsewhere herein.

In a Block/Redirect Step 1320 the communication of a blocked commandreceived from a member of clients is blocked or redirected and, thus,prevented from reaching a member of Game Logic 210. The blocked commandis optionally redirected to an add-on, alternative game code, and/or thelike. The blocked command is optionally then be replaced by an allowed,alternative command. In some embodiments, the blocked command isredirected to Discovery Logic 1250 configured to attempt to identify theblocked input control. When this occurs, the Discovery Logic 1250optionally attempts to identify the blocked input control on the graphicoutput that caused generation of the blocked command. If identified, theblocked input control can, thus, be blocked in the future usingtechniques described herein.

In Map Step 1170 part of a video overlay applied in Add Overlay Step1140 is mapped or remapped to an alternative game command or no gamecommand at all. Operation of the covered blocked input control istypically changed such that it does not generate a blocked command.Further details of this step are described elsewhere herein. Theidentification and redirection of a blocked input control in Steps 1320and 1170 is optionally performed automatically. The blocked inputcontrol can be a default part of a game or the result of an add-on.

In an optional Control Window Step 1325, a browser window or browser tabis opened or closed on Client 110A using Client Control Logic 1240.Control Window Step 1325 is optionally performed in response to theblocked or remapped input command received in Block/Redirect Step 1320.Block/Redirect Step 1320, Map Step 1170 and Control Window Step 1325 areoptionally performed independently of other steps illustrated in FIG.13. In some examples, the opened browser window is directed to a helpweb page, a registration or purchase web page, an add-on web page, anaccount management web page, a game selection web page, a user or add-onspecified web page, an alternative video stream including game video,and/or the like.

FIG. 14 illustrates methods of identifying blocked input controls,according to various embodiments of the invention. In these methods aregion of a game video frame is identified as including a blocked inputcontrol. The blocked input control can then be modified using an overlayin future video frames and/or the command generated by the blocked inputcontrol can be redirected or otherwise blocked.

In an Identify Step 1410, a blocked input command (a control command tobe blocked) is identified in commands received from a member of Clients110. For example, in various embodiments, the command is identifiedprior to or after being communicated to a member of Game Logic 210.Identification prior to receipt by Game Logic 210 typically requiresthat the command be recognized as one that could cause an undesirableresult. For example, a command known to be for accessing video settings.In some cases commands can only be identified as being blocked inputcommands based on what Game Logic 210 does in response to receiving thecommand. For example, if Game Logic 210A performs an undesirable fileaccess or attempts to modify a video driver after a command is received,then that command may automatically be considered a blocked inputcommand. Identify Step 1410 is optionally performed automatically byDiscovery Logic 1250. As described elsewhere herein, whether or not aninput command is a blocked input command is optionally state dependent.

In a Find Step 1415 a region (e.g., frame coordinates) within a videoframe that resulted in the blocked input command identified in IdentifyStep 1410 is found. This is optionally accomplished by identifying thelocation of a user “click” on the frame and/or by recognizing specificobjects, such as buttons or icons within the frame. Find Step 1415 isoptionally performed automatically using Command Processing Logic 1230,Sentinel Logic 1210 and/or Discovery Logic 1250.

In an optional Select Overlay Step 1420 an overlay is selected to modifythe region of future video frames found in Find Step 1415. The overlaymay modify all or part of a blocked input control depending on how muchof the input control has been found. More than one occurrence ofIdentify Step 1410 and Find Step 1415 may be required to fully identifya blocked input control. The overlay is optionally selected from a groupof standard overlays, from a “grayed out” version of the blocked inputcontrol, or from an image dynamically generated based on contents of thevideo frame. The selected overlay is configured to be applied to futureinstances of the video frame as monitored by Sentinel Logic 1210.

In an optional Store Select Overlay Step 1425 the selected overlay isstored in a location accessible to Overlay Logic 290. Store SelectOverlay Step 1425 is optionally in embodiments wherein the selectedoverlay is already stored in a location accessible to Overlay Logic 290.

In an optional Determine Mapping Step 1430 a mapping is determined fromthe modified input command control to an input command. For example, ablocked command control that resulted in sending of a video settingscommand may be remapped to alternative logic configured to offer allowedvideo operations. The blocked command control mayor may not be modifiedby an overlay. Remapping may be accomplished using Command ProcessingLogic 1230 and/or Discovery Logic 1250.

In an optional Store Mapping Step 1435 the mapping determined inDetermine Mapping Step 1430 is stored in a location accessible toCommand Processing Logic 1230.

Several embodiments are specifically illustrated and/or describedherein. However, it will be appreciated that modifications andvariations are covered by the above teachings and within the scope ofthe appended claims without departing from the spirit and intended scopethereof. For example, while the examples provided herein are directed tovideo games, the systems and methods discussed can be applied to otherinteractive video systems and other computer applications. For example,Game System 100 may be adapted to be a generalized application systemincluding application logic in place of Game Logic 210A and/or 210B.This application logic can include accounting applications, computeraided design applications, word processing applications, video on demandapplications, drawing applications, image editing or viewingapplications, browsing applications, communication applications,spreadsheet applications, database applications, presentation managementapplications, and/or the like. While the examples provided herein areprimarily directed at the delivery of video, the systems and methoddescribed herein can be applied to management of audio commands, audiooverlays, the generation of audio frames and delivery of audio streams.In some embodiments, audio and video streams are generated and deliveredtogether as part of an audio-video stream.

The embodiments discussed herein are illustrative of the presentinvention. As these embodiments of the present invention are describedwith reference to illustrations, various modifications or adaptations ofthe methods and or specific structures described may become apparent tothose skilled in the art. All such modifications, adaptations, orvariations that rely upon the teachings of the present invention, andthrough which these teachings have advanced the art, are considered tobe within the spirit and scope of the present invention. Hence, thesedescriptions and drawings should not be considered in a limiting sense,as it is understood that the present invention is in no way limited toonly the embodiments illustrated.

Computing systems referred to herein can comprise an integrated circuit,a microprocessor, a personal computer, a server, a distributed computingsystem, a communication device, a network device, or the like, andvarious combinations of the same. A computing system may also comprisevolatile and/or non-volatile memory such as random access memory (RAM),dynamic random access memory (DRAM), static random access memory (SRAM),magnetic media, optical media, nano-media, a hard drive, a compact disk,a digital versatile disc (DVD), and/or other devices configured forstoring analog or digital information, such as in a database. Thevarious examples of logic noted above can comprise hardware, firmware,or software stored on a computer readable medium, or combinationsthereof. A computer-readable medium, as used herein, expressly excludespaper. Computer-implemented steps of the methods noted herein cancomprise a set of instructions stored on a computer-readable medium thatwhen executed cause the computing system to perform the steps. Acomputing system programmed to perform particular functions pursuant toinstructions from program software is a special purpose computing systemfor performing those particular functions. Data that is manipulated by aspecial purpose computing system while performing those particularfunctions is at least electronically saved in buffers of the computingsystem, physically changing the special purpose computing system fromone state to the next with each change to the stored data.

The logic discussed herein may include hardware, firmware and/orsoftware stored on a computer readable medium. This logic may beimplemented in an electronic device to produce a special purposecomputing system.

What is claimed is:
 1. A video server system comprising: an applicationengine for executing a first computer application; a graphics processingunit for generating a graphic output in response to graphics datareceived from the first computer application and a point of view;command processing logic for blocking communication of an input commandto the first computer application by examining input commands receivedby one or more input/output devices before forwarding the input commandsto the first computer application, so as to affect the graphic output,wherein video frames of the graphic output is adjusted to not include aninput control for the input command that is blocked; an encodingpipeline for generating an encoded video stream using the graphicoutput, wherein the encoded video stream includes video frames that donot include the input control for the input command that is blocked bythe command processing logic; an output logic for communicating theencoded video stream to a remote client via a communication network, sothat the video frames rendered on the remote client do not graphicallydisplay the input control for the input command that is blocked; and amicroprocessor for executing the command processing logic at the videoserver system.
 2. The system of claim 1, wherein the command processinglogic selects specific input commands to block depending on a state ofthe first computer application.
 3. The system of claim 1, wherein thecommand processing logic determines specific input commands to blockbased on characteristics of a game player's account.
 4. The system ofclaim 1, wherein the command processing logic includes a second inputcontrol associated with a non-blocked input command in place of theinput control associated with the blocked input command, interaction atthe second input control re-directs the first computer application toload a pre-approved add-on on to the video server system.
 5. The systemof claim 1, wherein the command processing logic processes the inputcommand that is blocked by saving a state of the first computerapplication in static storage.
 6. The system of claim 1, wherein thecommand processing logic processes the input command that is blocked soas to activate an alternate application.
 7. The system of claim 1,wherein the command processing logic processes the input command that isblocked by engaging a client control logic to open or close a window ortab on the remote client, wherein the window or tab is a browser windowor browser tab.
 8. The system of claim 7, wherein the command processinglogic initiates generation of a video of a virtual file system usinginformation provided by the video server system, wherein the initiationincludes engaging the client control logic to open a second browserwindow for rendering images of the first computer application.
 9. Thesystem of claim 7, wherein the client control logic provides a uniformresource locator (URL) to a video output of the video server system whenthe browser window is opened, the video output providing a video of avirtual file system of the video server system, wherein navigation andselection of a file from the virtual file system is enabled usingvirtual controls provided within the video.
 10. The system of claim 7,wherein the client control logic loads a website into the browser windowor browser tab on the remote client, when the browser window or browsertab is opened.
 11. The system of claim 10, wherein the website is asocial networking website.
 12. The system of claim 1, wherein thecommand processing logic causes processing of the input command that isblocked by executing a second computer application on the remote client.13. The system of claim 1, wherein the command processing logicprocesses the input command that is blocked by passing a different inputcommand to the first computer application.
 14. The system of claim 1,further comprising state logic for changing operation of the commandprocessing logic responsive to a state of the first computerapplication, such that different sets of input commands are blockeddepending on the state.
 15. The system of claim 1, wherein the firstcomputer application is a video game.
 16. A method comprising: executinga first computer application using an application engine running on avideo server system; generating, by a graphics processor unit of thevideo server system, a graphic output in response to graphics datareceived from the first computer application and a point of view;examining, by a command processing logic of the video server system,input commands received by one or more input/output devices beforeforwarding the input commands to the first computer application toaffect the graphic output, wherein the examining includes blockingcommunication of an input command to the first computer application soas to not be included in the graphic output, wherein the blocking causesadjustment to video frames of the graphic output so as to not includeinput control for the input command that is blocked; generating, by anencoding pipeline of the video server system, an encoded video streamusing the graphic output, wherein the encoded video stream includesvideo frames that do not include the input control for the input commandthat is blocked; and communicating, by an output logic of the videoserver system, the encoded video stream to a remote client via acommunication network, so that the video frames rendered on the remoteclient do not graphically display the input control for the inputcommand that is blocked.
 17. The method of claim 16, wherein theexamining of input commands further includes identifying specific inputcommands that are blocked, based on a state of the first computerapplication.
 18. The method of claim 16, further includes directing thefirst computer application to load a pre-approved add-on on to the videoserver system, upon detecting the input command that is blocked.
 19. Themethod of claim 16, further includes saving a state of the firstcomputer application in static storage, upon detecting the input commandthat is blocked.
 20. The method of claim 16, further includes processingthe input command that is blocked by activating a client control logic,wherein the client control logic is configured to open or close a windowor tab on the remote client, wherein the window or tab is a browserwindow or browser tab.
 21. A non-transitory computer-readable mediumhaving program instructions, which when executed by a microprocessor ina video server system performs a method, the program instructionsinclude: program instructions for executing a first computerapplication; program instructions for generating a graphic output inresponse to graphics data received from the first computer applicationand a point of view; program instructions for examining input commandsreceived by one or more input/output devices before forwarding the inputcommands to the first computer application to affect the graphic output,wherein the program instructions for examining includes programinstructions for blocking communication of an input command that isblocked to the first computer application so as to not be included inthe graphic output, wherein the program instructions for blocking causesadjustment to video frames of the graphic output so as to not includeinput control for the input command that is blocked; programinstructions for generating an encoded video stream using the graphicoutput, wherein the encoded video stream includes video frames that donot include the input control for the input command that is blocked; andprogram instructions for communicating the encoded video stream to aremote client via a communication network, so that the video framesrendered on the remote client do not graphically display the inputcontrol for the input command that is blocked.